@redpanda-data/docs-extensions-and-macros 5.54.0 → 5.55.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/doc-tools-mcp.js +4 -5
- package/bin/doc-tools.js +39 -2
- package/bin/mcp-tools/generated-docs-review.js +26 -9
- package/bin/mcp-tools/overrides-audit.js +1 -1
- package/bin/mcp-tools/rpcn-docs.js +4 -14
- package/cli-utils/format-helm-spec.js +4 -3
- package/cli-utils/helm-commented-values.js +88 -18
- package/cli-utils/self-managed-docs-branch.js +36 -30
- package/extension-utils/html-utils.js +27 -0
- package/extensions/REFERENCE.adoc +20 -4
- package/extensions/add-git-dates.js +13 -1
- package/extensions/aggregate-terms.js +34 -20
- package/extensions/modify-connect-tag-playbook.js +181 -3
- package/extensions/resolve-xrefs-in-attachments.js +1 -12
- package/extensions/util/connect-docs-asset.js +6 -5
- package/extensions/version-fetcher/fetch-latest-docker-tag.js +35 -13
- package/extensions/version-fetcher/get-latest-redpanda-helm-version-from-operator.js +64 -40
- package/extensions/version-fetcher/list-connect-releases.js +3 -3
- package/extensions/version-fetcher/release-lines.js +64 -0
- package/extensions/version-fetcher/set-latest-version.js +73 -3
- package/macros/README.adoc +1 -0
- package/macros/REFERENCE.adoc +99 -0
- package/macros/badge.js +5 -8
- package/macros/enterprise.js +2 -5
- package/macros/glossary.js +4 -14
- package/macros/iceberg-explorer.js +242 -0
- package/macros/prop.js +3 -13
- package/mcp/USER_GUIDE.adoc +25 -135
- package/package.json +4 -2
- package/tools/doc-impact-recording/index.js +190 -0
- package/tools/overrides-audit/README.adoc +1 -2
- package/tools/overrides-audit/adapters/properties.js +1 -1
- package/tools/overrides-audit/index.js +3 -4
- package/tools/redpanda-connect/AUTOMATION.md +3 -36
- package/tools/redpanda-connect/README.adoc +5 -33
- package/tools/redpanda-connect/generate-rpcn-connector-docs.js +3 -3
- package/tools/redpanda-connect/pr-summary-formatter.js +6 -4
- package/tools/redpanda-connect/report-delta.js +6 -3
- package/tools/redpanda-connect/rpcn-connector-docs-handler.js +38 -16
- package/tools/redpanda-release-notes/generate-release-notes.js +116 -1
- package/tools/rpk-docs/templates/command.hbs +1 -1
- package/tools/overrides-audit/adapters/connect.js +0 -144
package/mcp/USER_GUIDE.adoc
CHANGED
|
@@ -200,24 +200,6 @@ The MCP server provides *writer-friendly tools* designed for natural language in
|
|
|
200
200
|
* Requires knowledge of doc-tools CLI syntax
|
|
201
201
|
* Only works in repos that have doc-tools installed
|
|
202
202
|
|
|
203
|
-
=== Contextual guides (prompts)
|
|
204
|
-
|
|
205
|
-
*property-docs-guide*:: Comprehensive guide for updating Redpanda property documentation
|
|
206
|
-
+
|
|
207
|
-
* Explains the auto-generation system and override workflow
|
|
208
|
-
* Provides rules for property descriptions (no cloud conditionals, no enterprise includes)
|
|
209
|
-
* Lists common scenarios and validation steps
|
|
210
|
-
* Claude automatically retrieves this when working with property documentation
|
|
211
|
-
* Ensures correct workflow: update property-overrides.json, not generated .adoc files
|
|
212
|
-
|
|
213
|
-
*rpcn-connector-docs-guide*:: Comprehensive guide for updating Redpanda Connect connector reference documentation
|
|
214
|
-
+
|
|
215
|
-
* Explains the override system with `$ref` syntax for DRY (Don't Repeat Yourself) principles
|
|
216
|
-
* Describes repository structure and how autogenerated content works with overrides
|
|
217
|
-
* Provides workflow for improving connector documentation
|
|
218
|
-
* Claude automatically retrieves this when working with connector documentation
|
|
219
|
-
* Ensures correct workflow: update docs-data/overrides.json, use $ref for deduplication
|
|
220
|
-
|
|
221
203
|
=== Background job management
|
|
222
204
|
|
|
223
205
|
Long-running documentation generation tasks support background execution with progress updates:
|
|
@@ -265,130 +247,38 @@ Background jobs send real-time progress notifications via MCP protocol, so Claud
|
|
|
265
247
|
|
|
266
248
|
All commands have a 10-minute timeout (increased from 5 minutes) to handle large documentation generation tasks.
|
|
267
249
|
|
|
268
|
-
==
|
|
269
|
-
|
|
270
|
-
The MCP server provides *AI consistency prompts* that help Claude follow your team's documentation standards automatically. These prompts are pre-configured instructions that guide Claude's responses for common documentation tasks.
|
|
271
|
-
|
|
272
|
-
=== Documentation review prompts
|
|
273
|
-
|
|
274
|
-
*review-for-style*:: Review documentation content for style guide compliance
|
|
275
|
-
+
|
|
276
|
-
* Checks: Style guide compliance, terminology consistency, voice/tone, AsciiDoc formatting
|
|
277
|
-
* Input: Content to review (can be file path or raw text)
|
|
278
|
-
* Output: Detailed feedback with critical issues, suggestions, and actionable recommendations
|
|
279
|
-
* Use when: Reviewing any documentation before publishing
|
|
280
|
-
|
|
281
|
-
*check-terminology*:: Validate terminology usage against approved team terminology
|
|
282
|
-
+
|
|
283
|
-
* Checks: Incorrect terms, deprecated terms (whitelist/blacklist, master/slave), inconsistent usage
|
|
284
|
-
* Input: Content to check
|
|
285
|
-
* Output: Critical issues with corrections, inconsistencies found, missing glossary links
|
|
286
|
-
* Use when: Ensuring consistent terminology across documentation
|
|
287
|
-
|
|
288
|
-
*improve-clarity*:: Refactor existing documentation for clarity and readability
|
|
289
|
-
+
|
|
290
|
-
* Applies: Style guide principles, simplifies complex explanations, improves structure
|
|
291
|
-
* Input: Content to improve
|
|
292
|
-
* Output: Refactored content with explanations of changes made
|
|
293
|
-
* Use when: Content is technically correct but difficult to understand
|
|
294
|
-
|
|
295
|
-
=== Content creation prompts
|
|
296
|
-
|
|
297
|
-
*write-new-guide*:: Generate a new tutorial or guide following team standards
|
|
298
|
-
+
|
|
299
|
-
* Applies: Team templates, approved terminology, voice/tone guidelines, AsciiDoc formatting
|
|
300
|
-
* Input: Topic (what to write about), optional audience (beginners, experienced devs, operators)
|
|
301
|
-
* Output: Complete guide following team standards
|
|
302
|
-
* Use when: Starting a new tutorial, how-to, or getting started guide
|
|
303
|
-
|
|
304
|
-
=== Workflow guide prompts
|
|
305
|
-
|
|
306
|
-
*property-docs-guide*:: Comprehensive guide for updating Redpanda property documentation
|
|
307
|
-
+
|
|
308
|
-
* Explains: Auto-generation system, override workflow, validation steps
|
|
309
|
-
* Rules: Never edit generated `.adoc` files, always update `property-overrides.json`
|
|
310
|
-
* Use when: Claude is helping with property documentation
|
|
311
|
-
* Claude automatically retrieves this prompt when working with properties
|
|
312
|
-
|
|
313
|
-
*rpcn-connector-docs-guide*:: Guide for updating Redpanda Connect connector documentation
|
|
314
|
-
+
|
|
315
|
-
* Explains: Override system with `$ref` syntax for DRY principles, repository structure
|
|
316
|
-
* Rules: Update `docs-data/overrides.json`, use `$ref` for deduplication
|
|
317
|
-
* Use when: Claude is helping with connector documentation
|
|
318
|
-
* Claude automatically retrieves this prompt when working with connectors
|
|
319
|
-
|
|
320
|
-
=== How to use prompts
|
|
250
|
+
== Guidance from docs-team-standards
|
|
321
251
|
|
|
322
|
-
|
|
252
|
+
This server provides tools and one resource. It has no MCP prompts. Writing, review, and workflow guidance lives in the https://github.com/redpanda-data/docs-team-standards[docs-team-standards^] Claude Code plugin:
|
|
323
253
|
|
|
324
|
-
|
|
254
|
+
[cols="1,2"]
|
|
255
|
+
|===
|
|
256
|
+
|Task |docs-team-standards
|
|
325
257
|
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
Claude: *retrieves and applies the review-for-style prompt*
|
|
329
|
-
*provides structured feedback following the prompt instructions*
|
|
330
|
-
----
|
|
331
|
-
|
|
332
|
-
*Implicit prompt usage:*
|
|
333
|
-
|
|
334
|
-
----
|
|
335
|
-
You: "Review this tutorial for our style guide"
|
|
336
|
-
Claude: *automatically uses review-for-style prompt*
|
|
337
|
-
*provides feedback based on your team standards*
|
|
338
|
-
----
|
|
339
|
-
|
|
340
|
-
*Listing available prompts:*
|
|
341
|
-
|
|
342
|
-
----
|
|
343
|
-
You: "List available prompts"
|
|
344
|
-
Claude: *shows all prompts with descriptions*
|
|
345
|
-
----
|
|
346
|
-
|
|
347
|
-
You can also preview prompts using the CLI:
|
|
258
|
+
|Review content for style, terminology, structure, and clarity
|
|
259
|
+
|The `review` skill, the `/review` command, or the `content-reviewer` agent
|
|
348
260
|
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
# Preview a review prompt
|
|
352
|
-
npx doc-tools preview-prompt review-for-style --content "Test content"
|
|
353
|
-
|
|
354
|
-
# Preview a write prompt
|
|
355
|
-
npx doc-tools preview-prompt write-new-guide --topic "Deploy cluster"
|
|
261
|
+
|Write a new page or guide
|
|
262
|
+
|The `page-templates` skill and the `/plan` command
|
|
356
263
|
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
264
|
+
|Update property documentation
|
|
265
|
+
|The `property-overrides` skill
|
|
266
|
+
|===
|
|
360
267
|
|
|
361
268
|
== Available resources
|
|
362
269
|
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
=== Style guide
|
|
366
|
-
|
|
367
|
-
*redpanda://style-guide*:: Redpanda Documentation Style Guide (v1.0.0)
|
|
270
|
+
*redpanda://personas*:: Target audience personas for the current repository
|
|
368
271
|
+
|
|
369
|
-
*
|
|
370
|
-
*
|
|
371
|
-
* Contains: Terminology quick reference, formatting examples, common patterns
|
|
372
|
-
* Claude automatically references this when reviewing or writing documentation
|
|
272
|
+
* Loaded from `docs-data/personas.yaml` in your current working directory
|
|
273
|
+
* Defines reader types, goals, pain points, and content preferences
|
|
373
274
|
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
Resources are automatically loaded by prompts, but you can also reference them explicitly:
|
|
377
|
-
|
|
378
|
-
*Explicit reference:*
|
|
275
|
+
To read it, ask Claude directly:
|
|
379
276
|
|
|
380
277
|
----
|
|
381
|
-
You: "Read the redpanda://
|
|
382
|
-
Claude: *reads the style guide resource*
|
|
383
|
-
*explains the code block formatting rules*
|
|
278
|
+
You: "Read the redpanda://personas resource and summarize the target readers"
|
|
384
279
|
----
|
|
385
280
|
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
----
|
|
389
|
-
You: "List available resources"
|
|
390
|
-
Claude: *shows all resources with descriptions and versions*
|
|
391
|
-
----
|
|
281
|
+
The style guide lives in docs-team-standards, not in this server.
|
|
392
282
|
|
|
393
283
|
=== Design philosophy
|
|
394
284
|
|
|
@@ -972,12 +862,12 @@ Suggestion: Check that the version exists in the Redpanda repository. You can fi
|
|
|
972
862
|
https://github.com/redpanda-data/streaming-enterprise/releases
|
|
973
863
|
----
|
|
974
864
|
|
|
975
|
-
=== Example 7: Property documentation updates
|
|
865
|
+
=== Example 7: Property documentation updates
|
|
976
866
|
|
|
977
867
|
----
|
|
978
868
|
You: Improve the description for cleanup.policy property to explain what each value does
|
|
979
869
|
|
|
980
|
-
Claude: *
|
|
870
|
+
Claude: *uses the property-overrides skill from docs-team-standards*
|
|
981
871
|
*reads property-overrides.json*
|
|
982
872
|
*updates the description field with enhanced explanation*
|
|
983
873
|
*uses generate_property_docs MCP tool with generate_partials: true*
|
|
@@ -995,7 +885,7 @@ The description now explains:
|
|
|
995
885
|
- `compact,delete` - Compaction + time-based deletion
|
|
996
886
|
----
|
|
997
887
|
|
|
998
|
-
|
|
888
|
+
The skill provides the property documentation workflow, so Claude updates the overrides file, not the generated files.
|
|
999
889
|
|
|
1000
890
|
=== Example 8: Review documentation for style compliance
|
|
1001
891
|
|
|
@@ -1004,7 +894,7 @@ You: Review this tutorial for style guide compliance:
|
|
|
1004
894
|
|
|
1005
895
|
[Content of tutorial pasted here]
|
|
1006
896
|
|
|
1007
|
-
Claude: *uses review-
|
|
897
|
+
Claude: *uses the review skill from docs-team-standards*
|
|
1008
898
|
*analyzes content against team style guide*
|
|
1009
899
|
|
|
1010
900
|
Style Review for Tutorial
|
|
@@ -1042,7 +932,7 @@ Overall: Good structure, clear explanations. Fix critical issues before publishi
|
|
|
1042
932
|
----
|
|
1043
933
|
You: Check the terminology in this doc for any deprecated or incorrect terms
|
|
1044
934
|
|
|
1045
|
-
Claude: *uses
|
|
935
|
+
Claude: *uses the review skill from docs-team-standards*
|
|
1046
936
|
|
|
1047
937
|
Terminology Review
|
|
1048
938
|
==========================================
|
|
@@ -1070,7 +960,7 @@ Summary:
|
|
|
1070
960
|
----
|
|
1071
961
|
You: Write a getting started guide for deploying a 3-node Redpanda cluster
|
|
1072
962
|
|
|
1073
|
-
Claude: *uses
|
|
963
|
+
Claude: *uses the page-templates skill from docs-team-standards*
|
|
1074
964
|
*applies team templates and standards*
|
|
1075
965
|
|
|
1076
966
|
= Deploy a Three-Node Redpanda Cluster
|
|
@@ -1103,7 +993,7 @@ You: This explanation of compaction is too complex. Can you simplify it?
|
|
|
1103
993
|
|
|
1104
994
|
[Paste complex explanation]
|
|
1105
995
|
|
|
1106
|
-
Claude: *uses
|
|
996
|
+
Claude: *uses the review skill from docs-team-standards*
|
|
1107
997
|
|
|
1108
998
|
Here's a clearer version:
|
|
1109
999
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@redpanda-data/docs-extensions-and-macros",
|
|
3
|
-
"version": "5.
|
|
3
|
+
"version": "5.55.0",
|
|
4
4
|
"description": "Antora extensions and macros developed for Redpanda documentation.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"antora",
|
|
@@ -94,7 +94,9 @@
|
|
|
94
94
|
"./extensions/property-page-index": "./extensions/property-page-index.js",
|
|
95
95
|
"./extensions/render-property-descriptions": "./extensions/render-property-descriptions.js",
|
|
96
96
|
"./extensions/kapa-source-groups": "./extensions/kapa-source-groups.js",
|
|
97
|
-
"./docs-data/kapa-source-groups.json": "./docs-data/kapa-source-groups.json"
|
|
97
|
+
"./docs-data/kapa-source-groups.json": "./docs-data/kapa-source-groups.json",
|
|
98
|
+
"./macros/iceberg-explorer": "./macros/iceberg-explorer.js",
|
|
99
|
+
"./macros/iceberg-explorer.js": "./macros/iceberg-explorer.js"
|
|
98
100
|
},
|
|
99
101
|
"files": [
|
|
100
102
|
"extensions",
|
|
@@ -0,0 +1,190 @@
|
|
|
1
|
+
'use strict'
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* Turn a Claude Code transcript into a doc-impact MCP recording.
|
|
5
|
+
*
|
|
6
|
+
* The doc-strings review asks the docs MCP server (docs.redpanda.com/mcp)
|
|
7
|
+
* whether published pages already cover a changed string. The doc-impact
|
|
8
|
+
* eval replays those answers per item, so an eval run sees the docs as they
|
|
9
|
+
* were when the review ran, not as they are today. A recording made later
|
|
10
|
+
* would already contain the writer's fix for the PR, and the pass would
|
|
11
|
+
* then correctly say "already documented" for a PR that needed docs.
|
|
12
|
+
*
|
|
13
|
+
* This reads the transcript the review step leaves behind and keeps only
|
|
14
|
+
* what a replay needs: the tool name, its arguments, and the result content.
|
|
15
|
+
* Nothing else from the transcript is copied, so the session's tokens,
|
|
16
|
+
* headers, environment and identity never reach the recording.
|
|
17
|
+
*
|
|
18
|
+
* The arguments are the model's own text, though, and the review reads
|
|
19
|
+
* untrusted PR content. A call whose arguments or result contain something
|
|
20
|
+
* shaped like a credential is dropped rather than saved, because the
|
|
21
|
+
* recording is uploaded as an artifact and log masking doesn't cover
|
|
22
|
+
* artifacts.
|
|
23
|
+
*
|
|
24
|
+
* Accepted input:
|
|
25
|
+
* - the execution file anthropics/claude-code-action writes: one JSON
|
|
26
|
+
* array of SDK messages;
|
|
27
|
+
* - `claude -p --output-format stream-json` output: one message per line.
|
|
28
|
+
* Both carry the same message shapes, so one parser serves both.
|
|
29
|
+
*/
|
|
30
|
+
|
|
31
|
+
const fs = require('fs')
|
|
32
|
+
const path = require('path')
|
|
33
|
+
|
|
34
|
+
const DEFAULT_SERVER = 'redpanda-docs'
|
|
35
|
+
|
|
36
|
+
// Credential shapes a recording must never carry: GitHub tokens (classic and
|
|
37
|
+
// fine-grained), JWTs (which include GitHub App and OIDC tokens), and AWS
|
|
38
|
+
// access key IDs.
|
|
39
|
+
const SECRET_PATTERNS = [
|
|
40
|
+
/\bgh[pousr]_[A-Za-z0-9]{20,}/,
|
|
41
|
+
/\bgithub_pat_[A-Za-z0-9_]{20,}/,
|
|
42
|
+
/\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{4,}/,
|
|
43
|
+
/\b(?:AKIA|ASIA)[0-9A-Z]{16}\b/
|
|
44
|
+
]
|
|
45
|
+
|
|
46
|
+
/** Whether a value, serialized, contains anything shaped like a credential. */
|
|
47
|
+
function looksSecret (value) {
|
|
48
|
+
const text = typeof value === 'string' ? value : JSON.stringify(value)
|
|
49
|
+
return SECRET_PATTERNS.some((re) => re.test(text || ''))
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
/** Messages from a transcript, in order. Lines that are not JSON are skipped. */
|
|
53
|
+
function parseTranscript (text) {
|
|
54
|
+
const raw = String(text || '')
|
|
55
|
+
const trimmed = raw.trim()
|
|
56
|
+
if (trimmed.startsWith('[')) {
|
|
57
|
+
try {
|
|
58
|
+
const all = JSON.parse(trimmed)
|
|
59
|
+
if (Array.isArray(all)) return all.filter((m) => m && typeof m === 'object')
|
|
60
|
+
} catch {
|
|
61
|
+
// Not one JSON document; read it line by line below.
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
const out = []
|
|
65
|
+
for (const line of raw.split('\n')) {
|
|
66
|
+
if (!line.trim()) continue
|
|
67
|
+
try {
|
|
68
|
+
const m = JSON.parse(line)
|
|
69
|
+
if (m && typeof m === 'object' && !Array.isArray(m)) out.push(m)
|
|
70
|
+
} catch {
|
|
71
|
+
// Progress lines and other non-JSON output are not messages.
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
return out
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
/** Tool-result content as an array of MCP content blocks. */
|
|
78
|
+
function contentBlocks (content) {
|
|
79
|
+
if (content == null) return []
|
|
80
|
+
if (typeof content === 'string') return [{ type: 'text', text: content }]
|
|
81
|
+
if (Array.isArray(content)) return content.map((b) => (typeof b === 'string' ? { type: 'text', text: b } : b))
|
|
82
|
+
return [{ type: 'text', text: JSON.stringify(content) }]
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
function blocksOf (message) {
|
|
86
|
+
const content = message && message.message && message.message.content
|
|
87
|
+
return Array.isArray(content) ? content : []
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
/**
|
|
91
|
+
* Every call to one MCP server's tools, paired with its result, in call
|
|
92
|
+
* order. A call with no result (the run stopped first) is left out: a
|
|
93
|
+
* replay of it would answer with nothing the server ever said.
|
|
94
|
+
*/
|
|
95
|
+
function extractCalls (messages, { server = DEFAULT_SERVER } = {}) {
|
|
96
|
+
const prefix = `mcp__${server}__`
|
|
97
|
+
const calls = []
|
|
98
|
+
const byId = new Map()
|
|
99
|
+
for (const m of messages || []) {
|
|
100
|
+
if (m.type === 'assistant') {
|
|
101
|
+
for (const b of blocksOf(m)) {
|
|
102
|
+
if (b && b.type === 'tool_use' && typeof b.name === 'string' && b.name.startsWith(prefix)) {
|
|
103
|
+
const call = { tool: b.name.slice(prefix.length), arguments: b.input || {}, content: null, is_error: false }
|
|
104
|
+
calls.push(call)
|
|
105
|
+
byId.set(b.id, call)
|
|
106
|
+
}
|
|
107
|
+
}
|
|
108
|
+
} else if (m.type === 'user') {
|
|
109
|
+
for (const b of blocksOf(m)) {
|
|
110
|
+
if (b && b.type === 'tool_result' && byId.has(b.tool_use_id)) {
|
|
111
|
+
const call = byId.get(b.tool_use_id)
|
|
112
|
+
call.content = contentBlocks(b.content)
|
|
113
|
+
call.is_error = Boolean(b.is_error)
|
|
114
|
+
}
|
|
115
|
+
}
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
return calls.filter((c) => c.content !== null)
|
|
119
|
+
}
|
|
120
|
+
|
|
121
|
+
/** Split calls into the ones safe to save and the count dropped as secret-bearing. */
|
|
122
|
+
function dropSecretCalls (calls) {
|
|
123
|
+
const kept = calls.filter((c) => !looksSecret(c.arguments) && !looksSecret(c.content))
|
|
124
|
+
return { kept, dropped: calls.length - kept.length }
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
/** The model the session ran on, from its init message. */
|
|
128
|
+
function sessionModel (messages) {
|
|
129
|
+
const init = (messages || []).find((m) => m.type === 'system' && m.subtype === 'init')
|
|
130
|
+
return (init && typeof init.model === 'string' && init.model) || null
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
/**
|
|
134
|
+
* A recording in the shape the doc-impact eval replays:
|
|
135
|
+
* { item, recorded_at, server_url, model, source, calls: [{ tool, arguments, content, is_error }] }
|
|
136
|
+
*/
|
|
137
|
+
function buildRecording (messages, { item = null, recordedAt, serverUrl = null, model = null, server = DEFAULT_SERVER } = {}) {
|
|
138
|
+
const { kept, dropped } = dropSecretCalls(extractCalls(messages, { server }))
|
|
139
|
+
const recording = {
|
|
140
|
+
item,
|
|
141
|
+
recorded_at: recordedAt || new Date().toISOString(),
|
|
142
|
+
server_url: serverUrl,
|
|
143
|
+
model: model || sessionModel(messages),
|
|
144
|
+
source: 'production',
|
|
145
|
+
calls: kept
|
|
146
|
+
}
|
|
147
|
+
// Not enumerable, so it never reaches the written file.
|
|
148
|
+
Object.defineProperty(recording, 'droppedSecretCalls', { value: dropped })
|
|
149
|
+
return recording
|
|
150
|
+
}
|
|
151
|
+
|
|
152
|
+
/**
|
|
153
|
+
* CLI entry: read --execution-file, write --output. Writes nothing when the
|
|
154
|
+
* session made no docs calls it can save, and removes any file an earlier
|
|
155
|
+
* run left at --output, so the caller can tell "no calls" from "a
|
|
156
|
+
* recording" by the file's presence. Exits 0 either way; a missing or
|
|
157
|
+
* unreadable transcript is exit 1.
|
|
158
|
+
*/
|
|
159
|
+
function runCli (options, { log = console.log, error = console.error } = {}) {
|
|
160
|
+
const file = options.executionFile
|
|
161
|
+
let text
|
|
162
|
+
try {
|
|
163
|
+
text = fs.readFileSync(file, 'utf8')
|
|
164
|
+
} catch (e) {
|
|
165
|
+
error(`Cannot read the transcript at ${file}: ${e.message}`)
|
|
166
|
+
return 1
|
|
167
|
+
}
|
|
168
|
+
const messages = parseTranscript(text)
|
|
169
|
+
const recording = buildRecording(messages, {
|
|
170
|
+
item: options.item || null,
|
|
171
|
+
recordedAt: options.recordedAt,
|
|
172
|
+
serverUrl: options.serverUrl || null,
|
|
173
|
+
model: options.model || null,
|
|
174
|
+
server: options.server || DEFAULT_SERVER
|
|
175
|
+
})
|
|
176
|
+
if (recording.droppedSecretCalls) {
|
|
177
|
+
error(`Dropped ${recording.droppedSecretCalls} MCP call(s) whose arguments or result look like a credential.`)
|
|
178
|
+
}
|
|
179
|
+
if (!recording.calls.length) {
|
|
180
|
+
fs.rmSync(options.output, { force: true })
|
|
181
|
+
log(`No ${options.server || DEFAULT_SERVER} MCP calls to save in ${messages.length} transcript message(s); no recording written.`)
|
|
182
|
+
return 0
|
|
183
|
+
}
|
|
184
|
+
fs.mkdirSync(path.dirname(path.resolve(options.output)), { recursive: true })
|
|
185
|
+
fs.writeFileSync(options.output, JSON.stringify(recording, null, 2) + '\n')
|
|
186
|
+
log(`Wrote ${recording.calls.length} ${options.server || DEFAULT_SERVER} MCP call(s) to ${options.output}.`)
|
|
187
|
+
return 0
|
|
188
|
+
}
|
|
189
|
+
|
|
190
|
+
module.exports = { parseTranscript, extractCalls, sessionModel, buildRecording, dropSecretCalls, looksSecret, runCli, DEFAULT_SERVER }
|
|
@@ -22,7 +22,7 @@ npx doc-tools overrides audit \
|
|
|
22
22
|
--format human
|
|
23
23
|
|
|
24
24
|
# Standalone
|
|
25
|
-
node tools/overrides-audit/index.js --overrides <path> --extracted <path> [--surface properties|rpk
|
|
25
|
+
node tools/overrides-audit/index.js --overrides <path> --extracted <path> [--surface properties|rpk] [--format json|human] [--output <file>]
|
|
26
26
|
----
|
|
27
27
|
|
|
28
28
|
IMPORTANT: `--extracted` must be the property extractor's *raw* output (the `--output` file), never the enhanced output (`--enhanced-output`) or the versioned `redpanda-properties-<tag>.json` attachments: those already have overrides applied, so every override would classify as REDUNDANT. The same trap exists for rpk: the versioned `rpk-v<version>.json` files in docs repos are post-override snapshots.
|
|
@@ -133,7 +133,6 @@ JSON (default): `{ surface, overrides_file, extracted_file, manifest, summary, c
|
|
|
133
133
|
|
|
134
134
|
* `properties` (complete): `property-overrides.json` vs the property extractor's raw JSON.
|
|
135
135
|
* `rpk` (structural): enumerates `rpk-overrides.json` (see `docs-data/rpk-overrides.schema.json`) -- command/flag `description` fields are the upstreamable prose (classified `REVIEW` with a TODO note until source comparison lands; it needs a raw pre-override `--print-tree` snapshot plus the `formatDescription()` pipeline from `tools/rpk-docs`), everything else (`seeAlso`, `pageAliases`, `content`, platform scoping, ...) is `KEEP` by design.
|
|
136
|
-
* `connect` (structural): enumerates the rp-connect-docs `overrides.json` (`definitions` + per-component arrays with nested `config.children`); prose fields classify `REVIEW` with a TODO note until `$ref` resolution and plugin-JSON comparison land.
|
|
137
136
|
|
|
138
137
|
== Golden run (local checkouts required)
|
|
139
138
|
|
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
* Reads docs-data/property-overrides.json and an extracted-properties JSON
|
|
5
5
|
* (the property extractor's raw --output, WITHOUT overrides applied) and
|
|
6
6
|
* classifies every override field. This is the complete reference adapter;
|
|
7
|
-
* rpk
|
|
7
|
+
* The rpk adapter follows the same shape.
|
|
8
8
|
*/
|
|
9
9
|
|
|
10
10
|
'use strict'
|
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
* node tools/overrides-audit/index.js \
|
|
13
13
|
* --overrides docs-data/property-overrides.json \
|
|
14
14
|
* --extracted tools/property-extractor/gen/properties-output.json \
|
|
15
|
-
* [--surface properties|rpk
|
|
15
|
+
* [--surface properties|rpk] [--format json|human] [--output <file>]
|
|
16
16
|
*
|
|
17
17
|
* Also exposed as `doc-tools overrides audit` and the `audit_overrides`
|
|
18
18
|
* MCP tool. See README.adoc in this directory for the classification rules
|
|
@@ -26,8 +26,7 @@ const path = require('path')
|
|
|
26
26
|
|
|
27
27
|
const ADAPTERS = {
|
|
28
28
|
properties: () => require('./adapters/properties'),
|
|
29
|
-
rpk: () => require('./adapters/rpk')
|
|
30
|
-
connect: () => require('./adapters/connect')
|
|
29
|
+
rpk: () => require('./adapters/rpk')
|
|
31
30
|
}
|
|
32
31
|
|
|
33
32
|
/**
|
|
@@ -153,7 +152,7 @@ function main (argv) {
|
|
|
153
152
|
.description('Classify docs-side override fields against extracted source strings')
|
|
154
153
|
.requiredOption('--overrides <path>', 'Path to the overrides JSON file (for example docs-data/property-overrides.json)')
|
|
155
154
|
.option('--extracted <path>', 'Path to the extracted source JSON (property extractor raw output; required for the properties surface)')
|
|
156
|
-
.option('--surface <surface>', 'Override surface: properties
|
|
155
|
+
.option('--surface <surface>', 'Override surface: properties or rpk', 'properties')
|
|
157
156
|
.option('--format <format>', 'Output format: json or human', 'json')
|
|
158
157
|
.option('--output <path>', 'Also write the JSON result to this file')
|
|
159
158
|
.action((options) => {
|
|
@@ -581,7 +581,7 @@ npx doc-tools generate rpcn-connector-docs \
|
|
|
581
581
|
| `--draft-missing` | Generate full page drafts for new connectors | false |
|
|
582
582
|
| `--update-whats-new` | Update whats-new.adoc with changes | false |
|
|
583
583
|
| `--include-bloblang` | Include Bloblang functions/methods | false |
|
|
584
|
-
| `--overrides <path>` | JSON file with description overrides |
|
|
584
|
+
| `--overrides <path>` | Optional JSON file with description overrides | None |
|
|
585
585
|
|
|
586
586
|
### Examples
|
|
587
587
|
|
|
@@ -766,41 +766,8 @@ Templates are in `tools/redpanda-connect/templates/`:
|
|
|
766
766
|
- `connector-examples.hbs` - Examples template
|
|
767
767
|
- `connector-full.hbs` - Full page draft template
|
|
768
768
|
|
|
769
|
-
###
|
|
770
|
-
|
|
771
|
-
|
|
772
|
-
```json
|
|
773
|
-
{
|
|
774
|
-
"inputs": {
|
|
775
|
-
"kafka": {
|
|
776
|
-
"fields": {
|
|
777
|
-
"addresses": {
|
|
778
|
-
"description": "Custom description override",
|
|
779
|
-
"examples": ["localhost:9092"]
|
|
780
|
-
}
|
|
781
|
-
}
|
|
782
|
-
}
|
|
783
|
-
}
|
|
784
|
-
}
|
|
785
|
-
```
|
|
786
|
-
|
|
787
|
-
Supports `$ref` syntax for deduplication:
|
|
788
|
-
```json
|
|
789
|
-
{
|
|
790
|
-
"inputs": {
|
|
791
|
-
"kafka": {
|
|
792
|
-
"fields": {
|
|
793
|
-
"tls": { "$ref": "#/common/tls" }
|
|
794
|
-
}
|
|
795
|
-
}
|
|
796
|
-
},
|
|
797
|
-
"common": {
|
|
798
|
-
"tls": {
|
|
799
|
-
"description": "TLS configuration (reused across components)"
|
|
800
|
-
}
|
|
801
|
-
}
|
|
802
|
-
}
|
|
803
|
-
```
|
|
769
|
+
### Fixing connector descriptions
|
|
770
|
+
Field, component, and Bloblang descriptions come from the component specs in `redpanda-data/connect` and `redpanda-data/benthos`. To fix wrong or missing text, open a PR against the spec. The fix ships in the next Connect release.
|
|
804
771
|
|
|
805
772
|
## Troubleshooting
|
|
806
773
|
|
|
@@ -46,11 +46,6 @@ npm install @redpanda-data/docs-extensions-and-macros
|
|
|
46
46
|
# Generate connector documentation with latest data
|
|
47
47
|
npx doc-tools generate rpcn-connector-docs --fetch-connectors
|
|
48
48
|
|
|
49
|
-
# Generate with custom overrides
|
|
50
|
-
npx doc-tools generate rpcn-connector-docs \
|
|
51
|
-
--fetch-connectors \
|
|
52
|
-
--overrides path/to/overrides.json
|
|
53
|
-
|
|
54
49
|
# Generate and draft missing connectors
|
|
55
50
|
npx doc-tools generate rpcn-connector-docs \
|
|
56
51
|
--fetch-connectors \
|
|
@@ -176,11 +171,6 @@ You can generate connector documentation with various options:
|
|
|
176
171
|
# Basic generation with fetching latest data
|
|
177
172
|
npx doc-tools generate rpcn-connector-docs --fetch-connectors
|
|
178
173
|
|
|
179
|
-
# Generate with custom overrides for descriptions/examples
|
|
180
|
-
npx doc-tools generate rpcn-connector-docs \
|
|
181
|
-
--fetch-connectors \
|
|
182
|
-
--overrides overrides.json
|
|
183
|
-
|
|
184
174
|
# Draft full pages for connectors missing documentation
|
|
185
175
|
npx doc-tools generate rpcn-connector-docs \
|
|
186
176
|
--fetch-connectors \
|
|
@@ -202,7 +192,7 @@ npx doc-tools generate rpcn-connector-docs \
|
|
|
202
192
|
* `--fetch-connectors` - Fetch latest data using `rpk connect list`
|
|
203
193
|
* `--connect-version <version>` - Specific Connect version to install and fetch (requires `--fetch-connectors`)
|
|
204
194
|
* `--draft-missing` - Generate full draft pages for undocumented connectors
|
|
205
|
-
* `--overrides <path>` -
|
|
195
|
+
* `--overrides <path>` - Merge descriptions and examples from an optional JSON file. Nothing is merged unless you pass it
|
|
206
196
|
* `--data-dir <path>` - Directory containing versioned connect JSON files
|
|
207
197
|
* `--old-data <path>` - Specific old version file for diff comparison
|
|
208
198
|
* `--include-bloblang` - Include Bloblang functions/methods
|
|
@@ -791,7 +781,9 @@ console.log('<!-- PR_SUMMARY_END -->');
|
|
|
791
781
|
|
|
792
782
|
=== Override system
|
|
793
783
|
|
|
794
|
-
|
|
784
|
+
Connector field, component, and Bloblang descriptions come from the component specs in https://github.com/redpanda-data/connect[redpanda-data/connect^] and https://github.com/redpanda-data/benthos[redpanda-data/benthos^]. To fix wrong or missing text, open a PR against the spec. The fix ships in the next Connect release.
|
|
785
|
+
|
|
786
|
+
The generator can also merge an optional JSON file passed with `--overrides` into the extracted data. Nothing is merged unless you pass the flag:
|
|
795
787
|
|
|
796
788
|
[,json]
|
|
797
789
|
----
|
|
@@ -840,11 +832,9 @@ Overrides support:
|
|
|
840
832
|
* Connector catalog enrichment
|
|
841
833
|
* Version tracking and change detection
|
|
842
834
|
* Handlebars template rendering
|
|
843
|
-
* Override system for custom documentation
|
|
844
835
|
|
|
845
836
|
=== Current limitations
|
|
846
837
|
|
|
847
|
-
* *Manual override maintenance*: Custom descriptions must be maintained in overrides.json
|
|
848
838
|
* *Example quality*: Auto-generated examples may not represent best practices
|
|
849
839
|
* *Duplicate YAML keys*: Some examples contain multiple configurations in one block, creating invalid YAML with duplicate top-level keys (for example, multiple `processors:` keys)
|
|
850
840
|
* *CSV catalog dependency*: Relies on manually maintained CSV in Connect repository
|
|
@@ -876,25 +866,7 @@ This occurs when multiple separate configurations are concatenated into a single
|
|
|
876
866
|
|
|
877
867
|
=== Workarounds
|
|
878
868
|
|
|
879
|
-
|
|
880
|
-
|
|
881
|
-
[,json]
|
|
882
|
-
----
|
|
883
|
-
{
|
|
884
|
-
"inputs": [
|
|
885
|
-
{
|
|
886
|
-
"name": "problematic_connector",
|
|
887
|
-
"description": "Custom description overriding auto-extracted text",
|
|
888
|
-
"examples": [
|
|
889
|
-
{
|
|
890
|
-
"title": "Corrected example",
|
|
891
|
-
"config": "input:\n problematic_connector:\n field: value"
|
|
892
|
-
}
|
|
893
|
-
]
|
|
894
|
-
}
|
|
895
|
-
]
|
|
896
|
-
}
|
|
897
|
-
----
|
|
869
|
+
To fix an unsupported pattern, a wrong description, or a broken example, edit the component spec in redpanda-data/connect or redpanda-data/benthos. The fix ships in the next Connect release.
|
|
898
870
|
|
|
899
871
|
== Contributing
|
|
900
872
|
|
|
@@ -101,9 +101,9 @@ function mergeOverrides(target, overrides) {
|
|
|
101
101
|
throw new Error('Target must be a valid object');
|
|
102
102
|
}
|
|
103
103
|
|
|
104
|
-
//
|
|
105
|
-
//
|
|
106
|
-
//
|
|
104
|
+
// Scalar keys are replaced outright. 'summary' must be listed: a key that
|
|
105
|
+
// falls through every branch below is silently dropped, and summaries feed
|
|
106
|
+
// page meta descriptions.
|
|
107
107
|
const scalarKeys = ['description', 'summary', 'type', 'annotated_field', 'version'];
|
|
108
108
|
|
|
109
109
|
for (const key in overrides) {
|