@redpanda-data/docs-extensions-and-macros 4.15.6 → 4.15.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/doc-tools.js +2 -0
- package/package.json +1 -1
- package/tools/redpanda-connect/AUTOMATION.md +834 -0
- package/tools/redpanda-connect/generate-rpcn-connector-docs.js +20 -1
- package/tools/redpanda-connect/github-release-utils.js +275 -0
- package/tools/redpanda-connect/helpers/buildConfigYaml.js +13 -8
- package/tools/redpanda-connect/multi-version-summary.js +92 -0
- package/tools/redpanda-connect/parse-csv-connectors.js +1 -0
- package/tools/redpanda-connect/pr-summary-formatter.js +381 -12
- package/tools/redpanda-connect/rpcn-connector-docs-handler.js +462 -66
- package/tools/redpanda-connect/templates/connector.hbs +1 -0
|
@@ -0,0 +1,834 @@
|
|
|
1
|
+
# Redpanda Connect Connector Documentation Automation
|
|
2
|
+
|
|
3
|
+
## Overview
|
|
4
|
+
|
|
5
|
+
This automation generates comprehensive reference documentation for Redpanda Connect connectors, including inputs, outputs, processors, buffers, caches, rate limiters, metrics, tracers, scanners, and optionally Bloblang functions/methods.
|
|
6
|
+
|
|
7
|
+
The automation handles **multi-release attribution**, automatically detecting and processing intermediate releases that may have been missed, ensuring that changes are accurately attributed to their actual release version rather than being lumped together.
|
|
8
|
+
|
|
9
|
+
## Goals
|
|
10
|
+
|
|
11
|
+
### Primary Goals
|
|
12
|
+
|
|
13
|
+
1. **Generate Comprehensive Reference Docs**: Create AsciiDoc documentation for all Redpanda Connect components with:
|
|
14
|
+
- Field descriptions with types, defaults, and options
|
|
15
|
+
- Working code examples (minimal and advanced configurations)
|
|
16
|
+
- Cross-references to related components
|
|
17
|
+
- Metadata (status badges: stable, beta, experimental, deprecated)
|
|
18
|
+
|
|
19
|
+
2. **Accurate Version Attribution**: Track when each component and field was introduced:
|
|
20
|
+
- Detect releases between the last documented version and latest
|
|
21
|
+
- Process each release pair sequentially
|
|
22
|
+
- Generate per-version change tracking
|
|
23
|
+
- Maintain historical accuracy even when releases are skipped
|
|
24
|
+
|
|
25
|
+
3. **Platform Support Detection**: Identify and document platform availability:
|
|
26
|
+
- **Cloud-supported**: Available in Redpanda Cloud (both serverless and BYOC)
|
|
27
|
+
- **Self-hosted only**: Available only in self-hosted deployments
|
|
28
|
+
- **Cloud-only**: Exclusive to Redpanda Cloud
|
|
29
|
+
- **Cgo-required**: Requires cgo-enabled builds
|
|
30
|
+
|
|
31
|
+
4. **Change Detection & Reporting**: Generate detailed change reports:
|
|
32
|
+
- New connectors and fields
|
|
33
|
+
- Removed/deprecated components
|
|
34
|
+
- Changed default values
|
|
35
|
+
- Breaking changes (removed fields)
|
|
36
|
+
|
|
37
|
+
## How It Works
|
|
38
|
+
|
|
39
|
+
### Architecture Overview
|
|
40
|
+
|
|
41
|
+
```text
|
|
42
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
43
|
+
│ 1. Version Detection │
|
|
44
|
+
│ • Read current version from antora.yml │
|
|
45
|
+
│ • Detect latest version from GitHub releases or rpk │
|
|
46
|
+
│ • Discover all intermediate releases │
|
|
47
|
+
└──────────────────────┬──────────────────────────────────────────┘
|
|
48
|
+
│
|
|
49
|
+
▼
|
|
50
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
51
|
+
│ 2. Sequential Release Processing (for each version pair) │
|
|
52
|
+
│ ┌───────────────────────────────────────────────────────┐ │
|
|
53
|
+
│ │ For each pair (v[n] → v[n+1]): │ │
|
|
54
|
+
│ │ a. Fetch connector data for both versions │ │
|
|
55
|
+
│ │ b. Run binary analysis (OSS, Cloud, cgo) │ │
|
|
56
|
+
│ │ c. Generate version-specific diff │ │
|
|
57
|
+
│ │ d. Track changes with version attribution │ │
|
|
58
|
+
│ └───────────────────────────────────────────────────────┘ │
|
|
59
|
+
└──────────────────────┬──────────────────────────────────────────┘
|
|
60
|
+
│
|
|
61
|
+
▼
|
|
62
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
63
|
+
│ 3. Final Version Processing │
|
|
64
|
+
│ • Generate AsciiDoc partials (fields & examples) │
|
|
65
|
+
│ • Create full page drafts for new connectors (optional) │
|
|
66
|
+
│ • Update navigation files │
|
|
67
|
+
│ • Create master diff aggregating all changes │
|
|
68
|
+
└──────────────────────┬──────────────────────────────────────────┘
|
|
69
|
+
│
|
|
70
|
+
▼
|
|
71
|
+
┌─────────────────────────────────────────────────────────────────┐
|
|
72
|
+
│ 4. Output Generation │
|
|
73
|
+
│ • Individual diffs: connect-diff-X.X.X_to_Y.Y.Y.json │
|
|
74
|
+
│ • Master diff: connect-diff-master-X.X.X_to_Z.Z.Z.json │
|
|
75
|
+
│ • PR summary with per-version attribution │
|
|
76
|
+
│ • Writer action items │
|
|
77
|
+
└─────────────────────────────────────────────────────────────────┘
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
### Multi-Release Processing
|
|
81
|
+
|
|
82
|
+
**Scenario**: antora.yml has version `4.50.0`, latest release is `4.54.0`
|
|
83
|
+
|
|
84
|
+
**Without Multi-Release (OLD behavior)**:
|
|
85
|
+
```text
|
|
86
|
+
4.50.0 ─────────────────────────► 4.54.0
|
|
87
|
+
(all changes lumped)
|
|
88
|
+
```
|
|
89
|
+
Result: All changes from 4.51.0, 4.52.0, 4.53.0, and 4.54.0 are attributed to 4.54.0 ❌
|
|
90
|
+
|
|
91
|
+
**With Multi-Release (NEW behavior)**:
|
|
92
|
+
```text
|
|
93
|
+
4.50.0 ──► 4.51.0 ──► 4.52.0 ──► 4.53.0 ──► 4.54.0
|
|
94
|
+
│ │ │ │ │
|
|
95
|
+
diff1 diff2 diff3 diff4 │
|
|
96
|
+
│ │ │ │ │
|
|
97
|
+
└──────────┴──────────┴──────────┴──────────┘
|
|
98
|
+
│
|
|
99
|
+
▼
|
|
100
|
+
master-diff.json
|
|
101
|
+
(accurate attribution)
|
|
102
|
+
```
|
|
103
|
+
Result: Each change is attributed to its actual release version ✅
|
|
104
|
+
|
|
105
|
+
### Version Detection Flow
|
|
106
|
+
|
|
107
|
+
1. **Determine Starting Version**:
|
|
108
|
+
- Read `asciidoc.attributes.latest-connect-version` from `antora.yml`
|
|
109
|
+
- OR use `--from-version` flag override
|
|
110
|
+
- OR fallback to latest JSON file in `docs-data/`
|
|
111
|
+
|
|
112
|
+
2. **Determine Target Version**:
|
|
113
|
+
- Use `--connect-version` flag (explicit version)
|
|
114
|
+
- OR auto-detect latest stable release from GitHub
|
|
115
|
+
- OR use local `rpk connect --version`
|
|
116
|
+
|
|
117
|
+
3. **Discover Intermediate Releases**:
|
|
118
|
+
- Query GitHub Releases API: `repos/redpanda-data/connect/releases`
|
|
119
|
+
- Filter to stable releases (exclude beta, RC, alpha)
|
|
120
|
+
- Parse semver and find all versions between start and target
|
|
121
|
+
- Sort chronologically
|
|
122
|
+
|
|
123
|
+
### Data Collection
|
|
124
|
+
|
|
125
|
+
For each version being processed:
|
|
126
|
+
|
|
127
|
+
1. **Connector Metadata** (via `rpk connect list`):
|
|
128
|
+
```json
|
|
129
|
+
{
|
|
130
|
+
"name": "kafka",
|
|
131
|
+
"type": "inputs",
|
|
132
|
+
"status": "stable",
|
|
133
|
+
"description": "...",
|
|
134
|
+
"summary": "...",
|
|
135
|
+
"config": { /* full schema */ }
|
|
136
|
+
}
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
2. **Binary Analysis** (download and inspect binaries):
|
|
140
|
+
- **OSS binary**: Standard self-hosted build
|
|
141
|
+
- **Cloud binary**: Redpanda Cloud serverless/BYOC build
|
|
142
|
+
- **Cgo binary**: Build with cgo-enabled components
|
|
143
|
+
|
|
144
|
+
Compares which connectors exist in each binary to determine:
|
|
145
|
+
- `inCloud`: Present in both OSS and Cloud
|
|
146
|
+
- `notInCloud`: Only in OSS (self-hosted only)
|
|
147
|
+
- `cloudOnly`: Only in Cloud binary
|
|
148
|
+
- `cgoOnly`: Only in cgo-enabled binary
|
|
149
|
+
|
|
150
|
+
3. **Metadata CSV** (optional, from GitHub):
|
|
151
|
+
- Commercial names for connectors
|
|
152
|
+
- Additional categorization info
|
|
153
|
+
|
|
154
|
+
### Change Detection
|
|
155
|
+
|
|
156
|
+
For each version pair `(oldVersion → newVersion)`:
|
|
157
|
+
|
|
158
|
+
```javascript
|
|
159
|
+
{
|
|
160
|
+
"comparison": {
|
|
161
|
+
"oldVersion": "4.50.0",
|
|
162
|
+
"newVersion": "4.51.0",
|
|
163
|
+
"timestamp": "2026-04-01T00:00:00.000Z"
|
|
164
|
+
},
|
|
165
|
+
"summary": {
|
|
166
|
+
"newComponents": 3, // New connectors
|
|
167
|
+
"newFields": 15, // New fields added to existing connectors
|
|
168
|
+
"removedComponents": 0,
|
|
169
|
+
"removedFields": 2, // Breaking changes!
|
|
170
|
+
"deprecatedComponents": 0,
|
|
171
|
+
"deprecatedFields": 1,
|
|
172
|
+
"changedDefaults": 0
|
|
173
|
+
},
|
|
174
|
+
"details": {
|
|
175
|
+
"newComponents": [
|
|
176
|
+
{
|
|
177
|
+
"name": "postgres_cdc",
|
|
178
|
+
"type": "inputs",
|
|
179
|
+
"status": "beta",
|
|
180
|
+
"version": "4.51.0", // Attribution!
|
|
181
|
+
"description": "..."
|
|
182
|
+
}
|
|
183
|
+
],
|
|
184
|
+
"newFields": [
|
|
185
|
+
{
|
|
186
|
+
"component": "inputs:kafka",
|
|
187
|
+
"field": "rack_id",
|
|
188
|
+
"description": "..."
|
|
189
|
+
}
|
|
190
|
+
],
|
|
191
|
+
// ... other change categories
|
|
192
|
+
},
|
|
193
|
+
"binaryAnalysis": {
|
|
194
|
+
"ossVersion": "4.51.0",
|
|
195
|
+
"cloudVersion": "4.52.0-rc1",
|
|
196
|
+
"comparison": {
|
|
197
|
+
"inCloud": [/*...*/],
|
|
198
|
+
"notInCloud": [/*...*/],
|
|
199
|
+
"cloudOnly": [/*...*/]
|
|
200
|
+
},
|
|
201
|
+
"cgoOnly": [/*...*/]
|
|
202
|
+
}
|
|
203
|
+
}
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
## Output Specifications
|
|
207
|
+
|
|
208
|
+
### 1. AsciiDoc Documentation Files
|
|
209
|
+
|
|
210
|
+
#### Field Partials (`modules/components/partials/fields/{type}/{name}.adoc`)
|
|
211
|
+
|
|
212
|
+
```asciidoc
|
|
213
|
+
// This content is autogenerated. Do not edit manually.
|
|
214
|
+
|
|
215
|
+
== Fields
|
|
216
|
+
|
|
217
|
+
=== `field_name`
|
|
218
|
+
|
|
219
|
+
Description of the field with details.
|
|
220
|
+
|
|
221
|
+
*Type*: `string`
|
|
222
|
+
|
|
223
|
+
*Default*: `"default_value"`
|
|
224
|
+
|
|
225
|
+
*Options*: `option1`, `option2`, `option3`
|
|
226
|
+
|
|
227
|
+
[source,yaml]
|
|
228
|
+
----
|
|
229
|
+
# Example:
|
|
230
|
+
field_name: example_value
|
|
231
|
+
----
|
|
232
|
+
|
|
233
|
+
=== `another_field`
|
|
234
|
+
|
|
235
|
+
...
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
**Requirements**:
|
|
239
|
+
- All fields documented with type, default, options
|
|
240
|
+
- Code examples in YAML
|
|
241
|
+
- Cross-references using `xref:` syntax
|
|
242
|
+
- Conditional content for deprecated/experimental fields
|
|
243
|
+
|
|
244
|
+
#### Example Partials (`modules/components/partials/examples/{type}/{name}.adoc`)
|
|
245
|
+
|
|
246
|
+
```asciidoc
|
|
247
|
+
// This content is autogenerated. Do not edit manually.
|
|
248
|
+
|
|
249
|
+
== Examples
|
|
250
|
+
|
|
251
|
+
=== Minimal configuration
|
|
252
|
+
|
|
253
|
+
Basic setup with required fields only
|
|
254
|
+
|
|
255
|
+
[source,yaml]
|
|
256
|
+
----
|
|
257
|
+
input:
|
|
258
|
+
kafka:
|
|
259
|
+
addresses: ["localhost:9092"]
|
|
260
|
+
topics: ["my_topic"]
|
|
261
|
+
----
|
|
262
|
+
|
|
263
|
+
=== Advanced configuration
|
|
264
|
+
|
|
265
|
+
Complete configuration with optional fields
|
|
266
|
+
|
|
267
|
+
[source,yaml]
|
|
268
|
+
----
|
|
269
|
+
input:
|
|
270
|
+
kafka:
|
|
271
|
+
addresses: ["localhost:9092"]
|
|
272
|
+
topics: ["my_topic"]
|
|
273
|
+
consumer_group: "my_group"
|
|
274
|
+
checkpoint_limit: 1000
|
|
275
|
+
# ... all fields
|
|
276
|
+
----
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
**Requirements**:
|
|
280
|
+
- Minimal example (required fields only)
|
|
281
|
+
- Advanced example (all meaningful fields)
|
|
282
|
+
- **Only output one example if they're identical** (no tabs needed)
|
|
283
|
+
- Use leading sentence: "Here's an example configuration:"
|
|
284
|
+
- Real-world, working configurations
|
|
285
|
+
|
|
286
|
+
#### Full Page Drafts (`modules/components/pages/{type}/{name}.adoc`)
|
|
287
|
+
|
|
288
|
+
Generated with `--draft-missing` flag for NEW connectors:
|
|
289
|
+
|
|
290
|
+
```asciidoc
|
|
291
|
+
= Connector Name
|
|
292
|
+
:type: input
|
|
293
|
+
:status: beta
|
|
294
|
+
:page-commercial-names: Commercial Name, Alternative Name
|
|
295
|
+
|
|
296
|
+
// tag::single-source[]
|
|
297
|
+
|
|
298
|
+
Brief summary of what this connector does.
|
|
299
|
+
|
|
300
|
+
== Common Use Cases
|
|
301
|
+
|
|
302
|
+
* Use case 1
|
|
303
|
+
* Use case 2
|
|
304
|
+
|
|
305
|
+
[tabs]
|
|
306
|
+
====
|
|
307
|
+
Common config::
|
|
308
|
+
+
|
|
309
|
+
include::redpanda-connect:components:partial$examples/inputs/connector_name.adoc[tag=common]
|
|
310
|
+
|
|
311
|
+
Advanced config::
|
|
312
|
+
+
|
|
313
|
+
include::redpanda-connect:components:partial$examples/inputs/connector_name.adoc[tag=advanced]
|
|
314
|
+
====
|
|
315
|
+
|
|
316
|
+
include::redpanda-connect:components:partial$fields/inputs/connector_name.adoc[]
|
|
317
|
+
|
|
318
|
+
// end::single-source[]
|
|
319
|
+
```
|
|
320
|
+
|
|
321
|
+
**Requirements**:
|
|
322
|
+
- Frontmatter with metadata
|
|
323
|
+
- Single-source tags for reuse in cloud docs
|
|
324
|
+
- Tabs for common vs advanced configs (only if different)
|
|
325
|
+
- Platform indicators (☁️ for cloud, 🔧 for cgo)
|
|
326
|
+
|
|
327
|
+
### 2. Data Files
|
|
328
|
+
|
|
329
|
+
#### Connector Data JSON (`docs-data/connect-{version}.json`)
|
|
330
|
+
|
|
331
|
+
Complete connector metadata for a specific version:
|
|
332
|
+
|
|
333
|
+
```json
|
|
334
|
+
{
|
|
335
|
+
"inputs": [
|
|
336
|
+
{
|
|
337
|
+
"name": "kafka",
|
|
338
|
+
"status": "stable",
|
|
339
|
+
"plugin": true,
|
|
340
|
+
"description": "...",
|
|
341
|
+
"summary": "...",
|
|
342
|
+
"config": {
|
|
343
|
+
"type": "object",
|
|
344
|
+
"fields": [
|
|
345
|
+
{
|
|
346
|
+
"name": "addresses",
|
|
347
|
+
"type": "array",
|
|
348
|
+
"description": "...",
|
|
349
|
+
"default": [],
|
|
350
|
+
"kind": "scalar"
|
|
351
|
+
}
|
|
352
|
+
]
|
|
353
|
+
},
|
|
354
|
+
"requiresCgo": false,
|
|
355
|
+
"cloudOnly": false
|
|
356
|
+
}
|
|
357
|
+
],
|
|
358
|
+
"outputs": [...],
|
|
359
|
+
"processors": [...],
|
|
360
|
+
// ... other component types
|
|
361
|
+
}
|
|
362
|
+
```
|
|
363
|
+
|
|
364
|
+
**Retention**: Only the latest version is kept after processing completes.
|
|
365
|
+
|
|
366
|
+
#### Individual Diff JSON (`docs-data/connect-diff-{v1}_to_{v2}.json`)
|
|
367
|
+
|
|
368
|
+
Changes between two consecutive versions:
|
|
369
|
+
|
|
370
|
+
```json
|
|
371
|
+
{
|
|
372
|
+
"comparison": {
|
|
373
|
+
"oldVersion": "4.50.0",
|
|
374
|
+
"newVersion": "4.51.0",
|
|
375
|
+
"timestamp": "2026-04-01T10:00:00.000Z"
|
|
376
|
+
},
|
|
377
|
+
"summary": {
|
|
378
|
+
"newComponents": 3,
|
|
379
|
+
"newFields": 15,
|
|
380
|
+
"removedFields": 2,
|
|
381
|
+
"deprecatedFields": 1
|
|
382
|
+
},
|
|
383
|
+
"details": {
|
|
384
|
+
"newComponents": [...],
|
|
385
|
+
"newFields": [...],
|
|
386
|
+
"removedFields": [...],
|
|
387
|
+
"deprecatedFields": [...],
|
|
388
|
+
"changedDefaults": [...]
|
|
389
|
+
},
|
|
390
|
+
"binaryAnalysis": {
|
|
391
|
+
"versions": {
|
|
392
|
+
"oss": "4.51.0",
|
|
393
|
+
"cloud": "4.52.0",
|
|
394
|
+
"cgo": "4.52.0"
|
|
395
|
+
},
|
|
396
|
+
"comparison": {
|
|
397
|
+
"inCloud": [...],
|
|
398
|
+
"notInCloud": [...],
|
|
399
|
+
"cloudOnly": [...]
|
|
400
|
+
},
|
|
401
|
+
"cgoOnly": [...],
|
|
402
|
+
"details": {
|
|
403
|
+
"cloudSupported": [...],
|
|
404
|
+
"selfHostedOnly": [...],
|
|
405
|
+
"cloudOnly": [...]
|
|
406
|
+
}
|
|
407
|
+
}
|
|
408
|
+
}
|
|
409
|
+
```
|
|
410
|
+
|
|
411
|
+
**Retention**: Kept for intermediate versions during processing, cleaned up after master diff is created.
|
|
412
|
+
|
|
413
|
+
#### Master Diff JSON (`docs-data/connect-diff-master-{v1}_to_{vN}.json`)
|
|
414
|
+
|
|
415
|
+
Aggregated changes across multiple releases:
|
|
416
|
+
|
|
417
|
+
```json
|
|
418
|
+
{
|
|
419
|
+
"metadata": {
|
|
420
|
+
"generatedAt": "2026-04-01T10:00:00.000Z",
|
|
421
|
+
"startVersion": "4.50.0",
|
|
422
|
+
"endVersion": "4.54.0",
|
|
423
|
+
"processedReleases": 4
|
|
424
|
+
},
|
|
425
|
+
"totalSummary": {
|
|
426
|
+
"versions": ["4.51.0", "4.52.0", "4.53.0", "4.54.0"],
|
|
427
|
+
"releaseCount": 4,
|
|
428
|
+
"newComponents": 12,
|
|
429
|
+
"newFields": 45,
|
|
430
|
+
"removedFields": 5,
|
|
431
|
+
"deprecatedFields": 3
|
|
432
|
+
},
|
|
433
|
+
"releases": [
|
|
434
|
+
{
|
|
435
|
+
"fromVersion": "4.50.0",
|
|
436
|
+
"toVersion": "4.51.0",
|
|
437
|
+
"date": "2024-05-01T00:00:00.000Z",
|
|
438
|
+
"summary": {...},
|
|
439
|
+
"details": {...},
|
|
440
|
+
"binaryAnalysis": {...}
|
|
441
|
+
},
|
|
442
|
+
{
|
|
443
|
+
"fromVersion": "4.51.0",
|
|
444
|
+
"toVersion": "4.52.0",
|
|
445
|
+
// ...
|
|
446
|
+
}
|
|
447
|
+
// ... one entry per release
|
|
448
|
+
]
|
|
449
|
+
}
|
|
450
|
+
```
|
|
451
|
+
|
|
452
|
+
**Purpose**: Provides writers with accurate per-version attribution for changelog/release notes.
|
|
453
|
+
|
|
454
|
+
### 3. PR Summary
|
|
455
|
+
|
|
456
|
+
Automatically generated PR description with platform indicators and action items:
|
|
457
|
+
|
|
458
|
+
```markdown
|
|
459
|
+
## 📊 Redpanda Connect Documentation Update
|
|
460
|
+
|
|
461
|
+
**📦 Multi-Release Update:** 4.50.0 → 4.54.0
|
|
462
|
+
**Releases Processed:** 4
|
|
463
|
+
**Cloud Version:** 4.55.0
|
|
464
|
+
|
|
465
|
+
### Total Changes Across All Releases
|
|
466
|
+
|
|
467
|
+
- **12** new connectors
|
|
468
|
+
- **45** new fields across 4 release(s)
|
|
469
|
+
- **5** removed fields ⚠️
|
|
470
|
+
- **3** deprecated fields
|
|
471
|
+
|
|
472
|
+
### Changes Per Release
|
|
473
|
+
|
|
474
|
+
#### 🔖 Version 4.51.0
|
|
475
|
+
|
|
476
|
+
**New Connectors (3):**
|
|
477
|
+
- `postgres_cdc` (inputs, beta) ☁️
|
|
478
|
+
- `tigerbeetle_cdc` (inputs, beta) 🔧
|
|
479
|
+
- `mongodb_cdc` (inputs, stable) ☁️
|
|
480
|
+
|
|
481
|
+
**New Fields:** 12 added
|
|
482
|
+
**⚠️ Removed Fields:** 2
|
|
483
|
+
|
|
484
|
+
#### 🔖 Version 4.52.0
|
|
485
|
+
|
|
486
|
+
**New Connectors (5):**
|
|
487
|
+
- `oracledb_cdc` (inputs, experimental) ☁️
|
|
488
|
+
- `elasticsearch_v9` (outputs, stable)
|
|
489
|
+
- ...
|
|
490
|
+
|
|
491
|
+
**New Fields:** 18 added
|
|
492
|
+
|
|
493
|
+
#### 🔖 Version 4.53.0
|
|
494
|
+
|
|
495
|
+
_No changes in this release_
|
|
496
|
+
|
|
497
|
+
#### 🔖 Version 4.54.0
|
|
498
|
+
|
|
499
|
+
**New Connectors (4):**
|
|
500
|
+
...
|
|
501
|
+
|
|
502
|
+
### ✍️ Writer Action Items
|
|
503
|
+
|
|
504
|
+
**Document New Connectors:**
|
|
505
|
+
|
|
506
|
+
- [ ] Document new `postgres_cdc` inputs from **4.51.0** ☁️
|
|
507
|
+
- [ ] Document new `tigerbeetle_cdc` inputs from **4.51.0** 🔧
|
|
508
|
+
- [ ] Document new `mongodb_cdc` inputs from **4.51.0** ☁️
|
|
509
|
+
- [ ] Document new `oracledb_cdc` inputs from **4.52.0** ☁️
|
|
510
|
+
- [ ] Document new `a2a_message` processors from **4.54.0** ☁️
|
|
511
|
+
|
|
512
|
+
### ☁️ Cloud Docs Update Required
|
|
513
|
+
|
|
514
|
+
**12** new connectors are available in Redpanda Cloud.
|
|
515
|
+
|
|
516
|
+
**Action:** Submit a separate PR to cloud-docs repository.
|
|
517
|
+
|
|
518
|
+
**For connectors in pages:**
|
|
519
|
+
\```asciidoc
|
|
520
|
+
include::redpanda-connect:components:page$type/name.adoc[tag=single-source]
|
|
521
|
+
\```
|
|
522
|
+
|
|
523
|
+
**For cloud-only connectors (in partials):**
|
|
524
|
+
\```asciidoc
|
|
525
|
+
include::redpanda-connect:components:partial$components/cloud-only/type/name.adoc[tag=single-source]
|
|
526
|
+
\```
|
|
527
|
+
|
|
528
|
+
### 🔧 Cgo Requirements
|
|
529
|
+
|
|
530
|
+
The following new connectors require cgo-enabled builds:
|
|
531
|
+
|
|
532
|
+
- `tigerbeetle_cdc` (inputs)
|
|
533
|
+
- `zmq4` (inputs, outputs)
|
|
534
|
+
- `ffi` (processors)
|
|
535
|
+
|
|
536
|
+
[Cgo installation instructions included]
|
|
537
|
+
|
|
538
|
+
<details>
|
|
539
|
+
<summary><strong>📋 Detailed Changes</strong> (click to expand)</summary>
|
|
540
|
+
|
|
541
|
+
[Comprehensive breakdown of all changes]
|
|
542
|
+
|
|
543
|
+
</details>
|
|
544
|
+
```
|
|
545
|
+
|
|
546
|
+
**Indicators**:
|
|
547
|
+
- ☁️ = Cloud-supported (available in Redpanda Cloud)
|
|
548
|
+
- 🔧 = Requires cgo-enabled build
|
|
549
|
+
- ⚠️ = Breaking change (removed fields)
|
|
550
|
+
|
|
551
|
+
## CLI Usage
|
|
552
|
+
|
|
553
|
+
### Basic Usage
|
|
554
|
+
|
|
555
|
+
```bash
|
|
556
|
+
# Generate docs for latest version
|
|
557
|
+
npx doc-tools generate rpcn-connector-docs --fetch-connectors
|
|
558
|
+
|
|
559
|
+
# Generate docs for specific version
|
|
560
|
+
npx doc-tools generate rpcn-connector-docs \
|
|
561
|
+
--fetch-connectors \
|
|
562
|
+
--connect-version 4.54.0
|
|
563
|
+
|
|
564
|
+
# Process intermediate releases with custom starting version
|
|
565
|
+
npx doc-tools generate rpcn-connector-docs \
|
|
566
|
+
--fetch-connectors \
|
|
567
|
+
--from-version 4.50.0 \
|
|
568
|
+
--connect-version 4.54.0
|
|
569
|
+
```
|
|
570
|
+
|
|
571
|
+
### CLI Flags
|
|
572
|
+
|
|
573
|
+
| Flag | Description | Default |
|
|
574
|
+
|------|-------------|---------|
|
|
575
|
+
| `--fetch-connectors` | Fetch fresh connector data using rpk | - |
|
|
576
|
+
| `--connect-version <version>` | Target Connect version to process | Auto-detect latest |
|
|
577
|
+
| `--from-version <version>` | Override starting version (instead of antora.yml) | Read from antora.yml |
|
|
578
|
+
| `--skip-intermediate` | Disable multi-release processing (legacy mode) | Multi-release enabled |
|
|
579
|
+
| `--cloud-version <version>` | Specific cloud binary version | Auto-detect latest |
|
|
580
|
+
| `--cgo-version <version>` | Specific cgo binary version | Same as cloud |
|
|
581
|
+
| `--draft-missing` | Generate full page drafts for new connectors | false |
|
|
582
|
+
| `--update-whats-new` | Update whats-new.adoc with changes | false |
|
|
583
|
+
| `--include-bloblang` | Include Bloblang functions/methods | false |
|
|
584
|
+
| `--overrides <path>` | JSON file with description overrides | `docs-data/overrides.json` |
|
|
585
|
+
|
|
586
|
+
### Examples
|
|
587
|
+
|
|
588
|
+
#### Catchup After Missed Releases
|
|
589
|
+
|
|
590
|
+
```bash
|
|
591
|
+
# antora.yml has 4.50.0, but latest is 4.54.0
|
|
592
|
+
# This will process all 4 intermediate releases
|
|
593
|
+
|
|
594
|
+
npx doc-tools generate rpcn-connector-docs \
|
|
595
|
+
--fetch-connectors \
|
|
596
|
+
--draft-missing \
|
|
597
|
+
--update-whats-new
|
|
598
|
+
```
|
|
599
|
+
|
|
600
|
+
**Output**:
|
|
601
|
+
- Processes: 4.50.0→4.51.0, 4.51.0→4.52.0, 4.52.0→4.53.0, 4.53.0→4.54.0
|
|
602
|
+
- Creates: 4 individual diffs + 1 master diff
|
|
603
|
+
- Generates: Partials, drafts, PR summary with per-version attribution
|
|
604
|
+
|
|
605
|
+
#### Legacy Single-Version Mode
|
|
606
|
+
|
|
607
|
+
```bash
|
|
608
|
+
# Disable multi-release processing (old behavior)
|
|
609
|
+
|
|
610
|
+
npx doc-tools generate rpcn-connector-docs \
|
|
611
|
+
--fetch-connectors \
|
|
612
|
+
--skip-intermediate
|
|
613
|
+
```
|
|
614
|
+
|
|
615
|
+
#### Custom Version Range
|
|
616
|
+
|
|
617
|
+
```bash
|
|
618
|
+
# Process releases between specific versions
|
|
619
|
+
|
|
620
|
+
npx doc-tools generate rpcn-connector-docs \
|
|
621
|
+
--fetch-connectors \
|
|
622
|
+
--from-version 4.52.0 \
|
|
623
|
+
--connect-version 4.54.0
|
|
624
|
+
```
|
|
625
|
+
|
|
626
|
+
## File Structure
|
|
627
|
+
|
|
628
|
+
```
|
|
629
|
+
docs-extensions-and-macros/
|
|
630
|
+
├── tools/redpanda-connect/
|
|
631
|
+
│ ├── rpcn-connector-docs-handler.js # Main orchestration
|
|
632
|
+
│ ├── generate-rpcn-connector-docs.js # Doc generation logic
|
|
633
|
+
│ ├── report-delta.js # Diff generation
|
|
634
|
+
│ ├── pr-summary-formatter.js # PR summary formatting
|
|
635
|
+
│ ├── github-release-utils.js # GitHub API integration
|
|
636
|
+
│ ├── multi-version-summary.js # Master diff aggregation
|
|
637
|
+
│ ├── connector-binary-analyzer.js # Binary download & analysis
|
|
638
|
+
│ └── update-whats-new.js # Release notes updates
|
|
639
|
+
│
|
|
640
|
+
├── docs-data/ # Generated data (gitignored)
|
|
641
|
+
│ ├── connect-{version}.json # Connector metadata (latest only)
|
|
642
|
+
│ ├── connect-diff-{v1}_to_{v2}.json # Individual diffs (intermediate)
|
|
643
|
+
│ └── connect-diff-master-{v1}_to_{vN}.json # Master diff (kept)
|
|
644
|
+
│
|
|
645
|
+
└── modules/components/
|
|
646
|
+
├── pages/{type}/{name}.adoc # Full pages (manually created or drafted)
|
|
647
|
+
└── partials/
|
|
648
|
+
├── fields/{type}/{name}.adoc # Auto-generated field docs
|
|
649
|
+
└── examples/{type}/{name}.adoc # Auto-generated examples
|
|
650
|
+
```
|
|
651
|
+
|
|
652
|
+
## Testing
|
|
653
|
+
|
|
654
|
+
### Unit Tests
|
|
655
|
+
|
|
656
|
+
```bash
|
|
657
|
+
# Run all tests
|
|
658
|
+
npm test
|
|
659
|
+
|
|
660
|
+
# Test specific modules
|
|
661
|
+
npm test -- __tests__/tools/github-release-utils.test.js
|
|
662
|
+
npm test -- __tests__/tools/pr-summary-formatter.test.js
|
|
663
|
+
```
|
|
664
|
+
|
|
665
|
+
**Coverage**:
|
|
666
|
+
- ✅ Version parsing and semver comparisons
|
|
667
|
+
- ✅ Prerelease filtering (beta/RC/alpha)
|
|
668
|
+
- ✅ Intermediate release discovery
|
|
669
|
+
- ✅ Platform detection (cloud vs self-hosted)
|
|
670
|
+
- ✅ PR summary formatting (single and multi-version)
|
|
671
|
+
- ✅ Diff generation and change detection
|
|
672
|
+
|
|
673
|
+
### Integration Testing
|
|
674
|
+
|
|
675
|
+
```bash
|
|
676
|
+
# Create test environment
|
|
677
|
+
mkdir -p /tmp/test-automation/{docs-data,modules/components/pages}
|
|
678
|
+
|
|
679
|
+
# Create mock antora.yml
|
|
680
|
+
echo 'name: test
|
|
681
|
+
version: main
|
|
682
|
+
asciidoc:
|
|
683
|
+
attributes:
|
|
684
|
+
latest-connect-version: "4.50.0"' > /tmp/test-automation/antora.yml
|
|
685
|
+
|
|
686
|
+
# Run automation
|
|
687
|
+
cd /tmp/test-automation
|
|
688
|
+
npx doc-tools generate rpcn-connector-docs \
|
|
689
|
+
--from-version 4.50.0 \
|
|
690
|
+
--connect-version 4.54.0 \
|
|
691
|
+
--fetch-connectors
|
|
692
|
+
```
|
|
693
|
+
|
|
694
|
+
**Verify**:
|
|
695
|
+
- ✅ Multiple diffs created (one per release pair)
|
|
696
|
+
- ✅ Master diff with accurate attribution
|
|
697
|
+
- ✅ AsciiDoc partials generated
|
|
698
|
+
- ✅ antora.yml updated to latest version
|
|
699
|
+
- ✅ Only latest JSON retained
|
|
700
|
+
|
|
701
|
+
## Dependencies
|
|
702
|
+
|
|
703
|
+
### Runtime Dependencies
|
|
704
|
+
- `@octokit/rest` - GitHub API client for release discovery
|
|
705
|
+
- `semver` - Semantic version parsing and comparison
|
|
706
|
+
- `handlebars` - Template engine for doc generation
|
|
707
|
+
- `js-yaml` - YAML parsing for antora.yml
|
|
708
|
+
|
|
709
|
+
### External Tools
|
|
710
|
+
- `rpk` - Redpanda CLI for fetching connector metadata
|
|
711
|
+
- `git` - For cloning repositories to fetch binary versions
|
|
712
|
+
|
|
713
|
+
## Error Handling
|
|
714
|
+
|
|
715
|
+
### GitHub API Rate Limiting
|
|
716
|
+
- **Without token**: 60 requests/hour
|
|
717
|
+
- **With token**: 5,000 requests/hour
|
|
718
|
+
- **Handling**: Cache responses, graceful degradation to single-version mode
|
|
719
|
+
|
|
720
|
+
### Missing Intermediate Data
|
|
721
|
+
- Automatically fetches from GitHub releases
|
|
722
|
+
- Downloads binaries for specified versions
|
|
723
|
+
- Falls back to rpk if available locally
|
|
724
|
+
|
|
725
|
+
### Network Failures
|
|
726
|
+
- Retries with exponential backoff
|
|
727
|
+
- Continues processing with partial data
|
|
728
|
+
- Logs warnings for manual review
|
|
729
|
+
|
|
730
|
+
## Edge Cases
|
|
731
|
+
|
|
732
|
+
### No Intermediate Releases
|
|
733
|
+
- Behaves like legacy single-version mode
|
|
734
|
+
- No master diff created
|
|
735
|
+
- Standard PR summary generated
|
|
736
|
+
|
|
737
|
+
### Beta/RC Versions
|
|
738
|
+
- Automatically filtered out
|
|
739
|
+
- Only stable GA releases processed
|
|
740
|
+
- Explicit override with `--include-prerelease` (not yet implemented)
|
|
741
|
+
|
|
742
|
+
### Identical Consecutive Versions
|
|
743
|
+
- Skips diff generation
|
|
744
|
+
- Logs "No changes detected"
|
|
745
|
+
- Updates metadata only
|
|
746
|
+
|
|
747
|
+
### Binary Unavailability
|
|
748
|
+
- Continues without binary analysis
|
|
749
|
+
- Platform indicators omitted from output
|
|
750
|
+
- Warning logged for manual verification
|
|
751
|
+
|
|
752
|
+
## Future Enhancements
|
|
753
|
+
|
|
754
|
+
1. **Bloblang Full Support**: Currently optional, could be made default
|
|
755
|
+
2. **Automated PR Creation**: Auto-submit PRs with generated content
|
|
756
|
+
3. **CI/CD Integration**: GitHub Actions workflow for weekly runs
|
|
757
|
+
4. **Historical Backfill**: Process all historical releases for complete attribution
|
|
758
|
+
5. **Diff Visualization**: Web UI to browse changes across versions
|
|
759
|
+
6. **Custom Templates**: User-provided Handlebars templates for docs
|
|
760
|
+
|
|
761
|
+
## Maintenance
|
|
762
|
+
|
|
763
|
+
### Updating Templates
|
|
764
|
+
Templates are in `tools/redpanda-connect/templates/`:
|
|
765
|
+
- `connector-fields.hbs` - Field documentation template
|
|
766
|
+
- `connector-examples.hbs` - Examples template
|
|
767
|
+
- `connector-full.hbs` - Full page draft template
|
|
768
|
+
|
|
769
|
+
### Updating Overrides
|
|
770
|
+
Override file: `docs-data/overrides.json` (or `--overrides` flag)
|
|
771
|
+
|
|
772
|
+
```json
|
|
773
|
+
{
|
|
774
|
+
"inputs": {
|
|
775
|
+
"kafka": {
|
|
776
|
+
"fields": {
|
|
777
|
+
"addresses": {
|
|
778
|
+
"description": "Custom description override",
|
|
779
|
+
"examples": ["localhost:9092"]
|
|
780
|
+
}
|
|
781
|
+
}
|
|
782
|
+
}
|
|
783
|
+
}
|
|
784
|
+
}
|
|
785
|
+
```
|
|
786
|
+
|
|
787
|
+
Supports `$ref` syntax for deduplication:
|
|
788
|
+
```json
|
|
789
|
+
{
|
|
790
|
+
"inputs": {
|
|
791
|
+
"kafka": {
|
|
792
|
+
"fields": {
|
|
793
|
+
"tls": { "$ref": "#/common/tls" }
|
|
794
|
+
}
|
|
795
|
+
}
|
|
796
|
+
},
|
|
797
|
+
"common": {
|
|
798
|
+
"tls": {
|
|
799
|
+
"description": "TLS configuration (reused across components)"
|
|
800
|
+
}
|
|
801
|
+
}
|
|
802
|
+
}
|
|
803
|
+
```
|
|
804
|
+
|
|
805
|
+
## Troubleshooting
|
|
806
|
+
|
|
807
|
+
### "No releases found in the specified range"
|
|
808
|
+
- **Cause**: Invalid version range or no releases exist between versions
|
|
809
|
+
- **Fix**: Verify versions exist on GitHub, check semver format
|
|
810
|
+
|
|
811
|
+
### "GitHub API rate limit exceeded"
|
|
812
|
+
- **Cause**: Too many API requests without authentication
|
|
813
|
+
- **Fix**: Set `GITHUB_TOKEN` environment variable
|
|
814
|
+
|
|
815
|
+
### "Binary analysis failed"
|
|
816
|
+
- **Cause**: Unable to download binaries (network, permissions, etc.)
|
|
817
|
+
- **Fix**: Check network, ensure write permissions to temp directories
|
|
818
|
+
|
|
819
|
+
### "Versions match, skipping diff"
|
|
820
|
+
- **Cause**: Already at target version, no work needed
|
|
821
|
+
- **Fix**: This is expected behavior, no action needed
|
|
822
|
+
|
|
823
|
+
## Contact & Support
|
|
824
|
+
|
|
825
|
+
For issues or questions:
|
|
826
|
+
- **GitHub Issues**: [docs-extensions-and-macros repository](https://github.com/redpanda-data/docs-extensions-and-macros/issues)
|
|
827
|
+
- **Slack**: #docs channel
|
|
828
|
+
- **Docs**: Internal Confluence documentation
|
|
829
|
+
|
|
830
|
+
---
|
|
831
|
+
|
|
832
|
+
**Last Updated**: 2026-04-01
|
|
833
|
+
**Version**: 1.0.0
|
|
834
|
+
**Maintainers**: Redpanda Docs Team
|