@redpanda-data/docs-extensions-and-macros 4.15.6 → 4.15.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,834 @@
1
+ # Redpanda Connect Connector Documentation Automation
2
+
3
+ ## Overview
4
+
5
+ This automation generates comprehensive reference documentation for Redpanda Connect connectors, including inputs, outputs, processors, buffers, caches, rate limiters, metrics, tracers, scanners, and optionally Bloblang functions/methods.
6
+
7
+ The automation handles **multi-release attribution**, automatically detecting and processing intermediate releases that may have been missed, ensuring that changes are accurately attributed to their actual release version rather than being lumped together.
8
+
9
+ ## Goals
10
+
11
+ ### Primary Goals
12
+
13
+ 1. **Generate Comprehensive Reference Docs**: Create AsciiDoc documentation for all Redpanda Connect components with:
14
+ - Field descriptions with types, defaults, and options
15
+ - Working code examples (minimal and advanced configurations)
16
+ - Cross-references to related components
17
+ - Metadata (status badges: stable, beta, experimental, deprecated)
18
+
19
+ 2. **Accurate Version Attribution**: Track when each component and field was introduced:
20
+ - Detect releases between the last documented version and latest
21
+ - Process each release pair sequentially
22
+ - Generate per-version change tracking
23
+ - Maintain historical accuracy even when releases are skipped
24
+
25
+ 3. **Platform Support Detection**: Identify and document platform availability:
26
+ - **Cloud-supported**: Available in Redpanda Cloud (both serverless and BYOC)
27
+ - **Self-hosted only**: Available only in self-hosted deployments
28
+ - **Cloud-only**: Exclusive to Redpanda Cloud
29
+ - **Cgo-required**: Requires cgo-enabled builds
30
+
31
+ 4. **Change Detection & Reporting**: Generate detailed change reports:
32
+ - New connectors and fields
33
+ - Removed/deprecated components
34
+ - Changed default values
35
+ - Breaking changes (removed fields)
36
+
37
+ ## How It Works
38
+
39
+ ### Architecture Overview
40
+
41
+ ```text
42
+ ┌─────────────────────────────────────────────────────────────────┐
43
+ │ 1. Version Detection │
44
+ │ • Read current version from antora.yml │
45
+ │ • Detect latest version from GitHub releases or rpk │
46
+ │ • Discover all intermediate releases │
47
+ └──────────────────────┬──────────────────────────────────────────┘
48
+
49
+
50
+ ┌─────────────────────────────────────────────────────────────────┐
51
+ │ 2. Sequential Release Processing (for each version pair) │
52
+ │ ┌───────────────────────────────────────────────────────┐ │
53
+ │ │ For each pair (v[n] → v[n+1]): │ │
54
+ │ │ a. Fetch connector data for both versions │ │
55
+ │ │ b. Run binary analysis (OSS, Cloud, cgo) │ │
56
+ │ │ c. Generate version-specific diff │ │
57
+ │ │ d. Track changes with version attribution │ │
58
+ │ └───────────────────────────────────────────────────────┘ │
59
+ └──────────────────────┬──────────────────────────────────────────┘
60
+
61
+
62
+ ┌─────────────────────────────────────────────────────────────────┐
63
+ │ 3. Final Version Processing │
64
+ │ • Generate AsciiDoc partials (fields & examples) │
65
+ │ • Create full page drafts for new connectors (optional) │
66
+ │ • Update navigation files │
67
+ │ • Create master diff aggregating all changes │
68
+ └──────────────────────┬──────────────────────────────────────────┘
69
+
70
+
71
+ ┌─────────────────────────────────────────────────────────────────┐
72
+ │ 4. Output Generation │
73
+ │ • Individual diffs: connect-diff-X.X.X_to_Y.Y.Y.json │
74
+ │ • Master diff: connect-diff-master-X.X.X_to_Z.Z.Z.json │
75
+ │ • PR summary with per-version attribution │
76
+ │ • Writer action items │
77
+ └─────────────────────────────────────────────────────────────────┘
78
+ ```
79
+
80
+ ### Multi-Release Processing
81
+
82
+ **Scenario**: antora.yml has version `4.50.0`, latest release is `4.54.0`
83
+
84
+ **Without Multi-Release (OLD behavior)**:
85
+ ```text
86
+ 4.50.0 ─────────────────────────► 4.54.0
87
+ (all changes lumped)
88
+ ```
89
+ Result: All changes from 4.51.0, 4.52.0, 4.53.0, and 4.54.0 are attributed to 4.54.0 ❌
90
+
91
+ **With Multi-Release (NEW behavior)**:
92
+ ```text
93
+ 4.50.0 ──► 4.51.0 ──► 4.52.0 ──► 4.53.0 ──► 4.54.0
94
+ │ │ │ │ │
95
+ diff1 diff2 diff3 diff4 │
96
+ │ │ │ │ │
97
+ └──────────┴──────────┴──────────┴──────────┘
98
+
99
+
100
+ master-diff.json
101
+ (accurate attribution)
102
+ ```
103
+ Result: Each change is attributed to its actual release version ✅
104
+
105
+ ### Version Detection Flow
106
+
107
+ 1. **Determine Starting Version**:
108
+ - Read `asciidoc.attributes.latest-connect-version` from `antora.yml`
109
+ - OR use `--from-version` flag override
110
+ - OR fallback to latest JSON file in `docs-data/`
111
+
112
+ 2. **Determine Target Version**:
113
+ - Use `--connect-version` flag (explicit version)
114
+ - OR auto-detect latest stable release from GitHub
115
+ - OR use local `rpk connect --version`
116
+
117
+ 3. **Discover Intermediate Releases**:
118
+ - Query GitHub Releases API: `repos/redpanda-data/connect/releases`
119
+ - Filter to stable releases (exclude beta, RC, alpha)
120
+ - Parse semver and find all versions between start and target
121
+ - Sort chronologically
122
+
123
+ ### Data Collection
124
+
125
+ For each version being processed:
126
+
127
+ 1. **Connector Metadata** (via `rpk connect list`):
128
+ ```json
129
+ {
130
+ "name": "kafka",
131
+ "type": "inputs",
132
+ "status": "stable",
133
+ "description": "...",
134
+ "summary": "...",
135
+ "config": { /* full schema */ }
136
+ }
137
+ ```
138
+
139
+ 2. **Binary Analysis** (download and inspect binaries):
140
+ - **OSS binary**: Standard self-hosted build
141
+ - **Cloud binary**: Redpanda Cloud serverless/BYOC build
142
+ - **Cgo binary**: Build with cgo-enabled components
143
+
144
+ Compares which connectors exist in each binary to determine:
145
+ - `inCloud`: Present in both OSS and Cloud
146
+ - `notInCloud`: Only in OSS (self-hosted only)
147
+ - `cloudOnly`: Only in Cloud binary
148
+ - `cgoOnly`: Only in cgo-enabled binary
149
+
150
+ 3. **Metadata CSV** (optional, from GitHub):
151
+ - Commercial names for connectors
152
+ - Additional categorization info
153
+
154
+ ### Change Detection
155
+
156
+ For each version pair `(oldVersion → newVersion)`:
157
+
158
+ ```javascript
159
+ {
160
+ "comparison": {
161
+ "oldVersion": "4.50.0",
162
+ "newVersion": "4.51.0",
163
+ "timestamp": "2026-04-01T00:00:00.000Z"
164
+ },
165
+ "summary": {
166
+ "newComponents": 3, // New connectors
167
+ "newFields": 15, // New fields added to existing connectors
168
+ "removedComponents": 0,
169
+ "removedFields": 2, // Breaking changes!
170
+ "deprecatedComponents": 0,
171
+ "deprecatedFields": 1,
172
+ "changedDefaults": 0
173
+ },
174
+ "details": {
175
+ "newComponents": [
176
+ {
177
+ "name": "postgres_cdc",
178
+ "type": "inputs",
179
+ "status": "beta",
180
+ "version": "4.51.0", // Attribution!
181
+ "description": "..."
182
+ }
183
+ ],
184
+ "newFields": [
185
+ {
186
+ "component": "inputs:kafka",
187
+ "field": "rack_id",
188
+ "description": "..."
189
+ }
190
+ ],
191
+ // ... other change categories
192
+ },
193
+ "binaryAnalysis": {
194
+ "ossVersion": "4.51.0",
195
+ "cloudVersion": "4.52.0-rc1",
196
+ "comparison": {
197
+ "inCloud": [/*...*/],
198
+ "notInCloud": [/*...*/],
199
+ "cloudOnly": [/*...*/]
200
+ },
201
+ "cgoOnly": [/*...*/]
202
+ }
203
+ }
204
+ ```
205
+
206
+ ## Output Specifications
207
+
208
+ ### 1. AsciiDoc Documentation Files
209
+
210
+ #### Field Partials (`modules/components/partials/fields/{type}/{name}.adoc`)
211
+
212
+ ```asciidoc
213
+ // This content is autogenerated. Do not edit manually.
214
+
215
+ == Fields
216
+
217
+ === `field_name`
218
+
219
+ Description of the field with details.
220
+
221
+ *Type*: `string`
222
+
223
+ *Default*: `"default_value"`
224
+
225
+ *Options*: `option1`, `option2`, `option3`
226
+
227
+ [source,yaml]
228
+ ----
229
+ # Example:
230
+ field_name: example_value
231
+ ----
232
+
233
+ === `another_field`
234
+
235
+ ...
236
+ ```
237
+
238
+ **Requirements**:
239
+ - All fields documented with type, default, options
240
+ - Code examples in YAML
241
+ - Cross-references using `xref:` syntax
242
+ - Conditional content for deprecated/experimental fields
243
+
244
+ #### Example Partials (`modules/components/partials/examples/{type}/{name}.adoc`)
245
+
246
+ ```asciidoc
247
+ // This content is autogenerated. Do not edit manually.
248
+
249
+ == Examples
250
+
251
+ === Minimal configuration
252
+
253
+ Basic setup with required fields only
254
+
255
+ [source,yaml]
256
+ ----
257
+ input:
258
+ kafka:
259
+ addresses: ["localhost:9092"]
260
+ topics: ["my_topic"]
261
+ ----
262
+
263
+ === Advanced configuration
264
+
265
+ Complete configuration with optional fields
266
+
267
+ [source,yaml]
268
+ ----
269
+ input:
270
+ kafka:
271
+ addresses: ["localhost:9092"]
272
+ topics: ["my_topic"]
273
+ consumer_group: "my_group"
274
+ checkpoint_limit: 1000
275
+ # ... all fields
276
+ ----
277
+ ```
278
+
279
+ **Requirements**:
280
+ - Minimal example (required fields only)
281
+ - Advanced example (all meaningful fields)
282
+ - **Only output one example if they're identical** (no tabs needed)
283
+ - Use leading sentence: "Here's an example configuration:"
284
+ - Real-world, working configurations
285
+
286
+ #### Full Page Drafts (`modules/components/pages/{type}/{name}.adoc`)
287
+
288
+ Generated with `--draft-missing` flag for NEW connectors:
289
+
290
+ ```asciidoc
291
+ = Connector Name
292
+ :type: input
293
+ :status: beta
294
+ :page-commercial-names: Commercial Name, Alternative Name
295
+
296
+ // tag::single-source[]
297
+
298
+ Brief summary of what this connector does.
299
+
300
+ == Common Use Cases
301
+
302
+ * Use case 1
303
+ * Use case 2
304
+
305
+ [tabs]
306
+ ====
307
+ Common config::
308
+ +
309
+ include::redpanda-connect:components:partial$examples/inputs/connector_name.adoc[tag=common]
310
+
311
+ Advanced config::
312
+ +
313
+ include::redpanda-connect:components:partial$examples/inputs/connector_name.adoc[tag=advanced]
314
+ ====
315
+
316
+ include::redpanda-connect:components:partial$fields/inputs/connector_name.adoc[]
317
+
318
+ // end::single-source[]
319
+ ```
320
+
321
+ **Requirements**:
322
+ - Frontmatter with metadata
323
+ - Single-source tags for reuse in cloud docs
324
+ - Tabs for common vs advanced configs (only if different)
325
+ - Platform indicators (☁️ for cloud, 🔧 for cgo)
326
+
327
+ ### 2. Data Files
328
+
329
+ #### Connector Data JSON (`docs-data/connect-{version}.json`)
330
+
331
+ Complete connector metadata for a specific version:
332
+
333
+ ```json
334
+ {
335
+ "inputs": [
336
+ {
337
+ "name": "kafka",
338
+ "status": "stable",
339
+ "plugin": true,
340
+ "description": "...",
341
+ "summary": "...",
342
+ "config": {
343
+ "type": "object",
344
+ "fields": [
345
+ {
346
+ "name": "addresses",
347
+ "type": "array",
348
+ "description": "...",
349
+ "default": [],
350
+ "kind": "scalar"
351
+ }
352
+ ]
353
+ },
354
+ "requiresCgo": false,
355
+ "cloudOnly": false
356
+ }
357
+ ],
358
+ "outputs": [...],
359
+ "processors": [...],
360
+ // ... other component types
361
+ }
362
+ ```
363
+
364
+ **Retention**: Only the latest version is kept after processing completes.
365
+
366
+ #### Individual Diff JSON (`docs-data/connect-diff-{v1}_to_{v2}.json`)
367
+
368
+ Changes between two consecutive versions:
369
+
370
+ ```json
371
+ {
372
+ "comparison": {
373
+ "oldVersion": "4.50.0",
374
+ "newVersion": "4.51.0",
375
+ "timestamp": "2026-04-01T10:00:00.000Z"
376
+ },
377
+ "summary": {
378
+ "newComponents": 3,
379
+ "newFields": 15,
380
+ "removedFields": 2,
381
+ "deprecatedFields": 1
382
+ },
383
+ "details": {
384
+ "newComponents": [...],
385
+ "newFields": [...],
386
+ "removedFields": [...],
387
+ "deprecatedFields": [...],
388
+ "changedDefaults": [...]
389
+ },
390
+ "binaryAnalysis": {
391
+ "versions": {
392
+ "oss": "4.51.0",
393
+ "cloud": "4.52.0",
394
+ "cgo": "4.52.0"
395
+ },
396
+ "comparison": {
397
+ "inCloud": [...],
398
+ "notInCloud": [...],
399
+ "cloudOnly": [...]
400
+ },
401
+ "cgoOnly": [...],
402
+ "details": {
403
+ "cloudSupported": [...],
404
+ "selfHostedOnly": [...],
405
+ "cloudOnly": [...]
406
+ }
407
+ }
408
+ }
409
+ ```
410
+
411
+ **Retention**: Kept for intermediate versions during processing, cleaned up after master diff is created.
412
+
413
+ #### Master Diff JSON (`docs-data/connect-diff-master-{v1}_to_{vN}.json`)
414
+
415
+ Aggregated changes across multiple releases:
416
+
417
+ ```json
418
+ {
419
+ "metadata": {
420
+ "generatedAt": "2026-04-01T10:00:00.000Z",
421
+ "startVersion": "4.50.0",
422
+ "endVersion": "4.54.0",
423
+ "processedReleases": 4
424
+ },
425
+ "totalSummary": {
426
+ "versions": ["4.51.0", "4.52.0", "4.53.0", "4.54.0"],
427
+ "releaseCount": 4,
428
+ "newComponents": 12,
429
+ "newFields": 45,
430
+ "removedFields": 5,
431
+ "deprecatedFields": 3
432
+ },
433
+ "releases": [
434
+ {
435
+ "fromVersion": "4.50.0",
436
+ "toVersion": "4.51.0",
437
+ "date": "2024-05-01T00:00:00.000Z",
438
+ "summary": {...},
439
+ "details": {...},
440
+ "binaryAnalysis": {...}
441
+ },
442
+ {
443
+ "fromVersion": "4.51.0",
444
+ "toVersion": "4.52.0",
445
+ // ...
446
+ }
447
+ // ... one entry per release
448
+ ]
449
+ }
450
+ ```
451
+
452
+ **Purpose**: Provides writers with accurate per-version attribution for changelog/release notes.
453
+
454
+ ### 3. PR Summary
455
+
456
+ Automatically generated PR description with platform indicators and action items:
457
+
458
+ ```markdown
459
+ ## 📊 Redpanda Connect Documentation Update
460
+
461
+ **📦 Multi-Release Update:** 4.50.0 → 4.54.0
462
+ **Releases Processed:** 4
463
+ **Cloud Version:** 4.55.0
464
+
465
+ ### Total Changes Across All Releases
466
+
467
+ - **12** new connectors
468
+ - **45** new fields across 4 release(s)
469
+ - **5** removed fields ⚠️
470
+ - **3** deprecated fields
471
+
472
+ ### Changes Per Release
473
+
474
+ #### 🔖 Version 4.51.0
475
+
476
+ **New Connectors (3):**
477
+ - `postgres_cdc` (inputs, beta) ☁️
478
+ - `tigerbeetle_cdc` (inputs, beta) 🔧
479
+ - `mongodb_cdc` (inputs, stable) ☁️
480
+
481
+ **New Fields:** 12 added
482
+ **⚠️ Removed Fields:** 2
483
+
484
+ #### 🔖 Version 4.52.0
485
+
486
+ **New Connectors (5):**
487
+ - `oracledb_cdc` (inputs, experimental) ☁️
488
+ - `elasticsearch_v9` (outputs, stable)
489
+ - ...
490
+
491
+ **New Fields:** 18 added
492
+
493
+ #### 🔖 Version 4.53.0
494
+
495
+ _No changes in this release_
496
+
497
+ #### 🔖 Version 4.54.0
498
+
499
+ **New Connectors (4):**
500
+ ...
501
+
502
+ ### ✍️ Writer Action Items
503
+
504
+ **Document New Connectors:**
505
+
506
+ - [ ] Document new `postgres_cdc` inputs from **4.51.0** ☁️
507
+ - [ ] Document new `tigerbeetle_cdc` inputs from **4.51.0** 🔧
508
+ - [ ] Document new `mongodb_cdc` inputs from **4.51.0** ☁️
509
+ - [ ] Document new `oracledb_cdc` inputs from **4.52.0** ☁️
510
+ - [ ] Document new `a2a_message` processors from **4.54.0** ☁️
511
+
512
+ ### ☁️ Cloud Docs Update Required
513
+
514
+ **12** new connectors are available in Redpanda Cloud.
515
+
516
+ **Action:** Submit a separate PR to cloud-docs repository.
517
+
518
+ **For connectors in pages:**
519
+ \```asciidoc
520
+ include::redpanda-connect:components:page$type/name.adoc[tag=single-source]
521
+ \```
522
+
523
+ **For cloud-only connectors (in partials):**
524
+ \```asciidoc
525
+ include::redpanda-connect:components:partial$components/cloud-only/type/name.adoc[tag=single-source]
526
+ \```
527
+
528
+ ### 🔧 Cgo Requirements
529
+
530
+ The following new connectors require cgo-enabled builds:
531
+
532
+ - `tigerbeetle_cdc` (inputs)
533
+ - `zmq4` (inputs, outputs)
534
+ - `ffi` (processors)
535
+
536
+ [Cgo installation instructions included]
537
+
538
+ <details>
539
+ <summary><strong>📋 Detailed Changes</strong> (click to expand)</summary>
540
+
541
+ [Comprehensive breakdown of all changes]
542
+
543
+ </details>
544
+ ```
545
+
546
+ **Indicators**:
547
+ - ☁️ = Cloud-supported (available in Redpanda Cloud)
548
+ - 🔧 = Requires cgo-enabled build
549
+ - ⚠️ = Breaking change (removed fields)
550
+
551
+ ## CLI Usage
552
+
553
+ ### Basic Usage
554
+
555
+ ```bash
556
+ # Generate docs for latest version
557
+ npx doc-tools generate rpcn-connector-docs --fetch-connectors
558
+
559
+ # Generate docs for specific version
560
+ npx doc-tools generate rpcn-connector-docs \
561
+ --fetch-connectors \
562
+ --connect-version 4.54.0
563
+
564
+ # Process intermediate releases with custom starting version
565
+ npx doc-tools generate rpcn-connector-docs \
566
+ --fetch-connectors \
567
+ --from-version 4.50.0 \
568
+ --connect-version 4.54.0
569
+ ```
570
+
571
+ ### CLI Flags
572
+
573
+ | Flag | Description | Default |
574
+ |------|-------------|---------|
575
+ | `--fetch-connectors` | Fetch fresh connector data using rpk | - |
576
+ | `--connect-version <version>` | Target Connect version to process | Auto-detect latest |
577
+ | `--from-version <version>` | Override starting version (instead of antora.yml) | Read from antora.yml |
578
+ | `--skip-intermediate` | Disable multi-release processing (legacy mode) | Multi-release enabled |
579
+ | `--cloud-version <version>` | Specific cloud binary version | Auto-detect latest |
580
+ | `--cgo-version <version>` | Specific cgo binary version | Same as cloud |
581
+ | `--draft-missing` | Generate full page drafts for new connectors | false |
582
+ | `--update-whats-new` | Update whats-new.adoc with changes | false |
583
+ | `--include-bloblang` | Include Bloblang functions/methods | false |
584
+ | `--overrides <path>` | JSON file with description overrides | `docs-data/overrides.json` |
585
+
586
+ ### Examples
587
+
588
+ #### Catchup After Missed Releases
589
+
590
+ ```bash
591
+ # antora.yml has 4.50.0, but latest is 4.54.0
592
+ # This will process all 4 intermediate releases
593
+
594
+ npx doc-tools generate rpcn-connector-docs \
595
+ --fetch-connectors \
596
+ --draft-missing \
597
+ --update-whats-new
598
+ ```
599
+
600
+ **Output**:
601
+ - Processes: 4.50.0→4.51.0, 4.51.0→4.52.0, 4.52.0→4.53.0, 4.53.0→4.54.0
602
+ - Creates: 4 individual diffs + 1 master diff
603
+ - Generates: Partials, drafts, PR summary with per-version attribution
604
+
605
+ #### Legacy Single-Version Mode
606
+
607
+ ```bash
608
+ # Disable multi-release processing (old behavior)
609
+
610
+ npx doc-tools generate rpcn-connector-docs \
611
+ --fetch-connectors \
612
+ --skip-intermediate
613
+ ```
614
+
615
+ #### Custom Version Range
616
+
617
+ ```bash
618
+ # Process releases between specific versions
619
+
620
+ npx doc-tools generate rpcn-connector-docs \
621
+ --fetch-connectors \
622
+ --from-version 4.52.0 \
623
+ --connect-version 4.54.0
624
+ ```
625
+
626
+ ## File Structure
627
+
628
+ ```
629
+ docs-extensions-and-macros/
630
+ ├── tools/redpanda-connect/
631
+ │ ├── rpcn-connector-docs-handler.js # Main orchestration
632
+ │ ├── generate-rpcn-connector-docs.js # Doc generation logic
633
+ │ ├── report-delta.js # Diff generation
634
+ │ ├── pr-summary-formatter.js # PR summary formatting
635
+ │ ├── github-release-utils.js # GitHub API integration
636
+ │ ├── multi-version-summary.js # Master diff aggregation
637
+ │ ├── connector-binary-analyzer.js # Binary download & analysis
638
+ │ └── update-whats-new.js # Release notes updates
639
+
640
+ ├── docs-data/ # Generated data (gitignored)
641
+ │ ├── connect-{version}.json # Connector metadata (latest only)
642
+ │ ├── connect-diff-{v1}_to_{v2}.json # Individual diffs (intermediate)
643
+ │ └── connect-diff-master-{v1}_to_{vN}.json # Master diff (kept)
644
+
645
+ └── modules/components/
646
+ ├── pages/{type}/{name}.adoc # Full pages (manually created or drafted)
647
+ └── partials/
648
+ ├── fields/{type}/{name}.adoc # Auto-generated field docs
649
+ └── examples/{type}/{name}.adoc # Auto-generated examples
650
+ ```
651
+
652
+ ## Testing
653
+
654
+ ### Unit Tests
655
+
656
+ ```bash
657
+ # Run all tests
658
+ npm test
659
+
660
+ # Test specific modules
661
+ npm test -- __tests__/tools/github-release-utils.test.js
662
+ npm test -- __tests__/tools/pr-summary-formatter.test.js
663
+ ```
664
+
665
+ **Coverage**:
666
+ - ✅ Version parsing and semver comparisons
667
+ - ✅ Prerelease filtering (beta/RC/alpha)
668
+ - ✅ Intermediate release discovery
669
+ - ✅ Platform detection (cloud vs self-hosted)
670
+ - ✅ PR summary formatting (single and multi-version)
671
+ - ✅ Diff generation and change detection
672
+
673
+ ### Integration Testing
674
+
675
+ ```bash
676
+ # Create test environment
677
+ mkdir -p /tmp/test-automation/{docs-data,modules/components/pages}
678
+
679
+ # Create mock antora.yml
680
+ echo 'name: test
681
+ version: main
682
+ asciidoc:
683
+ attributes:
684
+ latest-connect-version: "4.50.0"' > /tmp/test-automation/antora.yml
685
+
686
+ # Run automation
687
+ cd /tmp/test-automation
688
+ npx doc-tools generate rpcn-connector-docs \
689
+ --from-version 4.50.0 \
690
+ --connect-version 4.54.0 \
691
+ --fetch-connectors
692
+ ```
693
+
694
+ **Verify**:
695
+ - ✅ Multiple diffs created (one per release pair)
696
+ - ✅ Master diff with accurate attribution
697
+ - ✅ AsciiDoc partials generated
698
+ - ✅ antora.yml updated to latest version
699
+ - ✅ Only latest JSON retained
700
+
701
+ ## Dependencies
702
+
703
+ ### Runtime Dependencies
704
+ - `@octokit/rest` - GitHub API client for release discovery
705
+ - `semver` - Semantic version parsing and comparison
706
+ - `handlebars` - Template engine for doc generation
707
+ - `js-yaml` - YAML parsing for antora.yml
708
+
709
+ ### External Tools
710
+ - `rpk` - Redpanda CLI for fetching connector metadata
711
+ - `git` - For cloning repositories to fetch binary versions
712
+
713
+ ## Error Handling
714
+
715
+ ### GitHub API Rate Limiting
716
+ - **Without token**: 60 requests/hour
717
+ - **With token**: 5,000 requests/hour
718
+ - **Handling**: Cache responses, graceful degradation to single-version mode
719
+
720
+ ### Missing Intermediate Data
721
+ - Automatically fetches from GitHub releases
722
+ - Downloads binaries for specified versions
723
+ - Falls back to rpk if available locally
724
+
725
+ ### Network Failures
726
+ - Retries with exponential backoff
727
+ - Continues processing with partial data
728
+ - Logs warnings for manual review
729
+
730
+ ## Edge Cases
731
+
732
+ ### No Intermediate Releases
733
+ - Behaves like legacy single-version mode
734
+ - No master diff created
735
+ - Standard PR summary generated
736
+
737
+ ### Beta/RC Versions
738
+ - Automatically filtered out
739
+ - Only stable GA releases processed
740
+ - Explicit override with `--include-prerelease` (not yet implemented)
741
+
742
+ ### Identical Consecutive Versions
743
+ - Skips diff generation
744
+ - Logs "No changes detected"
745
+ - Updates metadata only
746
+
747
+ ### Binary Unavailability
748
+ - Continues without binary analysis
749
+ - Platform indicators omitted from output
750
+ - Warning logged for manual verification
751
+
752
+ ## Future Enhancements
753
+
754
+ 1. **Bloblang Full Support**: Currently optional, could be made default
755
+ 2. **Automated PR Creation**: Auto-submit PRs with generated content
756
+ 3. **CI/CD Integration**: GitHub Actions workflow for weekly runs
757
+ 4. **Historical Backfill**: Process all historical releases for complete attribution
758
+ 5. **Diff Visualization**: Web UI to browse changes across versions
759
+ 6. **Custom Templates**: User-provided Handlebars templates for docs
760
+
761
+ ## Maintenance
762
+
763
+ ### Updating Templates
764
+ Templates are in `tools/redpanda-connect/templates/`:
765
+ - `connector-fields.hbs` - Field documentation template
766
+ - `connector-examples.hbs` - Examples template
767
+ - `connector-full.hbs` - Full page draft template
768
+
769
+ ### Updating Overrides
770
+ Override file: `docs-data/overrides.json` (or `--overrides` flag)
771
+
772
+ ```json
773
+ {
774
+ "inputs": {
775
+ "kafka": {
776
+ "fields": {
777
+ "addresses": {
778
+ "description": "Custom description override",
779
+ "examples": ["localhost:9092"]
780
+ }
781
+ }
782
+ }
783
+ }
784
+ }
785
+ ```
786
+
787
+ Supports `$ref` syntax for deduplication:
788
+ ```json
789
+ {
790
+ "inputs": {
791
+ "kafka": {
792
+ "fields": {
793
+ "tls": { "$ref": "#/common/tls" }
794
+ }
795
+ }
796
+ },
797
+ "common": {
798
+ "tls": {
799
+ "description": "TLS configuration (reused across components)"
800
+ }
801
+ }
802
+ }
803
+ ```
804
+
805
+ ## Troubleshooting
806
+
807
+ ### "No releases found in the specified range"
808
+ - **Cause**: Invalid version range or no releases exist between versions
809
+ - **Fix**: Verify versions exist on GitHub, check semver format
810
+
811
+ ### "GitHub API rate limit exceeded"
812
+ - **Cause**: Too many API requests without authentication
813
+ - **Fix**: Set `GITHUB_TOKEN` environment variable
814
+
815
+ ### "Binary analysis failed"
816
+ - **Cause**: Unable to download binaries (network, permissions, etc.)
817
+ - **Fix**: Check network, ensure write permissions to temp directories
818
+
819
+ ### "Versions match, skipping diff"
820
+ - **Cause**: Already at target version, no work needed
821
+ - **Fix**: This is expected behavior, no action needed
822
+
823
+ ## Contact & Support
824
+
825
+ For issues or questions:
826
+ - **GitHub Issues**: [docs-extensions-and-macros repository](https://github.com/redpanda-data/docs-extensions-and-macros/issues)
827
+ - **Slack**: #docs channel
828
+ - **Docs**: Internal Confluence documentation
829
+
830
+ ---
831
+
832
+ **Last Updated**: 2026-04-01
833
+ **Version**: 1.0.0
834
+ **Maintainers**: Redpanda Docs Team