@redpanda-data/docs-extensions-and-macros 4.12.3 → 4.12.5

package/tools/property-extractor/transformers.py

@@ -80,24 +80,45 @@ Redpanda configuration properties are C++ objects with constructor signatures th
 based on feature requirements. Understanding these "arities" (parameter counts) is crucial
 for correctly extracting property metadata.
 
+┌─────────────────────────────────────────────────────────────────────────────
+│ ALL properties have a runtime mutability flag as their FIRST parameter
+│ (after *this which is skipped by the parser). This boolean indicates whether
+│ the property can be changed at runtime without restarting Redpanda.
+│
+│ C++ signature: property<T>(*this, runtime_mutable, name, description, ...)
+│ After parsing:  params[0] = runtime_mutable, params[1] = name, ...
+│                 params[-1] = default OR validator (validator is skipped)
+│
+│ Note: Some properties include validators as the last parameter. These are
+│       automatically detected and skipped when extracting the default value.
+│       The default is the last NON-VALIDATOR parameter.
+└─────────────────────────────────────────────────────────────────────────────
+
 BASIC PROPERTY PATTERNS:
 ┌─────────────────────────────────────────────────────────────────────────────
-│ 2-PARAMETER: property<T>(name, description)
-│ Example: property<bool>(*this, "enable_feature", "Enable the feature")
+│ 3-PARAMETER: property<T>(runtime_mutable, name, description)
+│ Example: property<bool>(*this, true, "enable_feature", "Enable the feature")
 │ Used for: Simple properties with no metadata or custom defaults
-│ Extraction: [0] = name, [1] = description
+│ Extraction: [0] = runtime_mutable, [1] = name, [2] = description
 └─────────────────────────────────────────────────────────────────────────────
 ┌─────────────────────────────────────────────────────────────────────────────
-│ 3-PARAMETER: property<T>(name, description, default)
-│ Example: property<int>(*this, "port", "Server port", 9092)
+│ 4-PARAMETER: property<T>(runtime_mutable, name, description, default)
+│ Example: property<int>(*this, false, "port", "Server port", 9092)
 │ Used for: Properties with simple custom default values
-│ Extraction: [0] = name, [1] = description, [2] = default
+│ Extraction: [0] = runtime_mutable, [1] = name, [2] = description, [3] = default
 └─────────────────────────────────────────────────────────────────────────────
 ┌─────────────────────────────────────────────────────────────────────────────
-│ 4-PARAMETER: property<T>(name, description, meta, default)
-│ Example: property<bool>(*this, "flag", "Description", meta{.needs_restart=yes}, true)
+│ 5-PARAMETER: property<T>(runtime_mutable, name, description, meta, default)
+│ Example: property<bool>(*this, true, "flag", "Desc", meta{.needs_restart=yes}, true)
 │ Used for: Properties with metadata (restart requirements, visibility, etc.)
-│ Extraction: [0] = name, [1] = description, [2] = meta{}, [3] = default
+│ Extraction: [0] = runtime_mutable, [1] = name, [2] = description, [3] = meta{}, [4] = default
+└─────────────────────────────────────────────────────────────────────────────
+┌─────────────────────────────────────────────────────────────────────────────
+│ WITH VALIDATOR: property<T>(runtime_mutable, name, description, [meta], default, validator)
+│ Example: property<int>(*this, true, "port", "Port", 9092, [](int v){return v > 0;})
+│ Used for: Properties with runtime validation functions
+│ Extraction: Validator is automatically skipped; default is found by searching backwards
+│ Note: Validators detected by is_validator_param() and excluded from default extraction
 └─────────────────────────────────────────────────────────────────────────────
 
 ENTERPRISE PROPERTY PATTERNS (More Complex):
@@ -517,6 +538,51 @@ def get_meta_value(info, key, default=None):
     return val
 
 
+def resolve_cpp_literal(value_str):
+    """
+    Resolve C++ literal expressions to their actual values.
+
+    Handles:
+    - Size literals: 5_MiB → 5242880
+    - std::to_string() wrapper: std::to_string(5_MiB) → 5242880
+    - Plain integers: 42 → 42
+    - Plain floats: 3.14 → 3.14
+
+    Args:
+        value_str: String containing C++ literal expression
+
+    Returns:
+        Resolved value (int, float, or original string if can't resolve)
+    """
+    if not isinstance(value_str, str):
+        return value_str
+
+    val = value_str.strip()
+
+    # Handle std::to_string() wrapper
+    to_string_match = re.match(r"std::to_string\((.+)\)", val)
+    if to_string_match:
+        val = to_string_match.group(1).strip()
+
+    # Try size literals like 5_MiB
+    size_match = re.match(r"(\d+)_([KMGTP])iB", val)
+    if size_match:
+        num, unit = int(size_match.group(1)), size_match.group(2)
+        mult = {"K": 1024, "M": 1024**2, "G": 1024**3, "T": 1024**4, "P": 1024**5}[unit]
+        return num * mult
+
+    # Try plain integer
+    if re.fullmatch(r"-?\d+", val):
+        return int(val)
+
+    # Try plain float
+    if re.fullmatch(r"-?\d+\.\d+", val):
+        return float(val)
+
+    # Return original if can't resolve
+    return value_str
+
+
 def get_resolve_constexpr_identifier():
     """
     Lazily import constexpr identifier resolution function to avoid circular imports.
@@ -1391,31 +1457,37 @@ class SimpleDefaultValuesTransformer:
             return property
 
         # Find where the meta{} param is
+        # MetaParamTransformer converts meta strings to dicts with "type": "initializer_list"
         meta_index = next(
             (i for i, p in enumerate(params)
              if isinstance(p.get("value"), (dict, str))
              and ("meta{" in str(p["value"]) or
-                  (isinstance(p["value"], dict) and "needs_restart" in p["value"]))),
-            None,
+                  (isinstance(p["value"], dict) and p["value"].get("type") == "initializer_list"))
+            ),
+            None
         )
 
-        # Default comes immediately after meta
-        if meta_index is None:
-            default_index = 3 if len(params) > 3 else None
-        else:
-            default_index = meta_index + 1 if len(params) > meta_index + 1 else None
+        # The default value is the LAST NON-VALIDATOR parameter in the constructor
+        # Property structure: (*this, runtime_mutability_flag, name, description, [meta], default, [validator])
+        # After *this is skipped: [0]=runtime_flag, [1]=name, [2]=desc, [3]=meta?, [4]=default, [5]=validator?
+        # We need to search backwards to find the last parameter that's not a validator
+        if len(params) < 2:
+            return property
 
-        if default_index is None or default_index >= len(params):
+        # Search backwards for the last non-validator parameter
+        default_index = None
+        for i in range(len(params) - 1, -1, -1):
+            if not is_validator_param(params[i]):
+                default_index = i
+                break
+
+        if default_index is None:
             return property
 
         # Candidate default param
         default_param = params[default_index]
         default = default_param.get("value")
 
-        # Skip obvious validator params
-        if is_validator_param(default_param):
-            return property
-
         # std::nullopt means "no default"
         if isinstance(default, str) and "std::nullopt" in default:
             property["default"] = None
@@ -1938,9 +2010,12 @@ class EnterpriseTransformer:
             info["params"] = params[:-1]
 
         # --- restricted_with_sanctioned (scalar form) ---
+        # Pattern: (restricted, sanctioned, name, description, meta, default)
+        # Must check that params[1] is NOT the property name (which would make it restricted_only)
         elif (
             len(params) >= 6
             and all(p["type"] in ("true", "false", "integer_literal", "string_literal", "qualified_identifier") for p in params[:2])
+            and self._clean_value(params[1]["value"]) != info.get("name_in_file")
         ):
             enterprise_constructor = "restricted_with_sanctioned"
             restricted_vals = [self._clean_value(params[0]["value"])]
@@ -1955,13 +2030,15 @@ class EnterpriseTransformer:
             restricted_vals = [self._clean_value(params[0]["value"])]
             info["params"] = params[1:]
 
-        # --- simple enterprise property (lambda validator pattern) ---
-        elif (len(params) >= 3 and 
-              params[0].get("type") == "lambda_expression" and
+        # --- simple enterprise property (function/lambda validator pattern) ---
+        # Pattern: (validator_function, name, description, meta, default, ...)
+        # The validator can be a lambda or a function identifier
+        elif (len(params) >= 3 and
+              params[0].get("type") in ("lambda_expression", "identifier", "qualified_identifier") and
               params[1].get("type") == "string_literal"):
             enterprise_constructor = "simple"
             # Don't modify params for simple enterprise properties - they have normal structure
-            # Remove the lambda validator from parameters as it's not needed for documentation
+            # Remove the validator from parameters as it's not needed for documentation
             info["params"] = params[1:]
 
         if not enterprise_constructor:
@@ -1980,28 +2057,6 @@ class EnterpriseTransformer:
         if sanctioned_vals is not None:
             property["enterprise_sanctioned_value"] = sanctioned_vals
 
-        # Add friendly description (values are already cleaned by _clean_value)
-        if enterprise_constructor == "restricted_with_sanctioned":
-            r = restricted_vals[0]
-            s = sanctioned_vals[0]
-            property["enterprise_default_description"] = (
-                f"Default: `{s}` (Community) or `{r}` (Enterprise)"
-            )
-        elif enterprise_constructor == "restricted_only":
-            if len(restricted_vals) > 1:
-                vals = ", ".join(f"`{v}`" for v in restricted_vals)
-                property["enterprise_default_description"] = (
-                    f"Available only with Enterprise license: {vals}"
-                )
-            else:
-                property["enterprise_default_description"] = (
-                    f"Available only with Enterprise license: `{restricted_vals[0]}`"
-                )
-        elif enterprise_constructor == "sanctioned_only":
-            property["enterprise_default_description"] = (
-                f"Community-only configuration. Sanctioned value: `{sanctioned_vals[0]}`"
-            )
-
         return property
 
     # --- Helper: clean literal/identifier values ---
@@ -2107,6 +2162,8 @@ class MetaParamTransformer:
 
                 key, value = [s.strip() for s in field.split("=", 1)]
                 clean_key = key.replace(".", "")
+                # Strip trailing commas from value (last field in meta block has trailing comma)
+                value = value.rstrip(",").strip()
                 meta_dict[clean_key] = value
 
                 # 🔹 Inline special handlers for known meta keys
@@ -2115,13 +2172,12 @@ class MetaParamTransformer:
                 # Example values
                 if clean_key == "example":
                     val_clean = value.strip().strip('"')
-                    # Try to coerce to int or float, else leave as string
-                    if re.fullmatch(r"-?\d+", val_clean):
-                        property["example"] = int(val_clean)
-                    elif re.fullmatch(r"-?\d+\.\d+", val_clean):
-                        property["example"] = float(val_clean)
+                    resolved = resolve_cpp_literal(val_clean)
+                    # Wrap in backticks for inline code formatting (handles overrides with AsciiDoc blocks)
+                    if isinstance(resolved, str):
+                        property["example"] = f"`{normalize_string(resolved)}`"
                     else:
-                        property["example"] = normalize_string(val_clean)
+                        property["example"] = f"`{resolved}`"
 
                 # Gets_restored / restored flags
                 elif clean_key in ("gets_restored", "restored"):
@@ -2204,21 +2260,18 @@ class ExampleTransformer:
                 meta_dict = param["value"]
                 if "example" in meta_dict:
                     example_val = meta_dict["example"]
-                    
+
                     # Clean up the value (remove quotes, etc.)
                     if isinstance(example_val, str):
                         example_val = example_val.strip().strip('"\'')
-                    
-                    # Try to coerce to appropriate type
-                    if isinstance(example_val, str):
-                        if re.fullmatch(r"-?\d+", example_val):
-                            property["example"] = int(example_val)
-                        elif re.fullmatch(r"-?\d+\.\d+", example_val):
-                            property["example"] = float(example_val)
-                        else:
-                            property["example"] = example_val
+
+                    # Use shared resolution logic
+                    resolved = resolve_cpp_literal(example_val)
+                    # Wrap in backticks for inline code formatting
+                    if isinstance(resolved, str):
+                        property["example"] = f"`{normalize_string(resolved)}`"
                     else:
-                        property["example"] = example_val
+                        property["example"] = f"`{resolved}`"
                     break
                     
         return property

package/tools/property-extractor/COMPUTED_CONSTANTS.md

@@ -1,173 +0,0 @@
-# Computed C++ Constants Resolution
-
-## Overview
-
-Some C++ constants in Redpanda are defined with complex compile-time expressions that cannot be easily parsed by the property-extractor. These constants need to be pre-computed and mapped to their actual values.
-
-## Problem Statement
-
-Properties like `log_message_timestamp_before_max_ms` use constants like `max_serializable_ms` as their default values. These constants are defined with complex expressions:
-
-```cpp
-// From src/v/serde/rw/chrono.h:20
-inline constexpr auto max_serializable_ms
-  = std::chrono::duration_cast<std::chrono::milliseconds>(
-    std::chrono::nanoseconds::max());
-```
-
-Without resolution, the extracted schema would show:
-```json
-{
-  "log_message_timestamp_before_max_ms": {
-    "default": "max_serializable_ms"  // ❌ String instead of numeric value
-  }
-}
-```
-
-## Solution
-
-### 1. COMPUTED_CONSTANTS Dictionary
-
-Added a dictionary in `transformers.py` that maps constant names to their computed values:
-
-```python
-COMPUTED_CONSTANTS = {
-    # From src/v/serde/rw/chrono.h:20
-    # inline constexpr auto max_serializable_ms = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::nanoseconds::max());
-    # Calculation: std::numeric_limits<int64_t>::max() / 1,000,000 = 9223372036854775807 / 1000000 = 9223372036854 ms
-    "max_serializable_ms": 9223372036854,  # ~292 years in milliseconds
-}
-```
-
-### 2. FriendlyDefaultTransformer Enhancement
-
-Updated the `FriendlyDefaultTransformer` to check the `COMPUTED_CONSTANTS` dictionary before falling back to string normalization:
-
-```python
-# ------------------------------------------------------------------
-# Computed C++ constants (max_serializable_ms, etc.)
-# ------------------------------------------------------------------
-if d in COMPUTED_CONSTANTS:
-    property["default"] = COMPUTED_CONSTANTS[d]
-    return property
-```
-
-### 3. Test Coverage
-
-Added comprehensive test in `tests/test_known_values.py`:
-
-```python
-def test_max_serializable_ms_constant_resolution(self):
-    """Test that max_serializable_ms constant is resolved to actual numeric value"""
-    info = create_complete_property_info(
-        name="log_message_timestamp_before_max_ms",
-        description="Maximum timestamp difference for record validation",
-        declaration="property<std::chrono::milliseconds> log_message_timestamp_before_max_ms;",
-        metadata="meta{.needs_restart = needs_restart::no, .visibility = visibility::user}",
-        default_value="max_serializable_ms"
-    )
-
-    property = apply_transformer_pipeline(info)
-
-    self.assertEqual(property["name"], "log_message_timestamp_before_max_ms")
-    self.assertEqual(property["type"], "integer")
-    # max_serializable_ms = std::numeric_limits<int64_t>::max() / 1,000,000 = 9223372036854 ms
-    self.assertEqual(property["default"], 9223372036854)
-    self.assertFalse(property["needs_restart"])
-    self.assertEqual(property["visibility"], "user")
-```
-
-## Calculation Details
-
-### max_serializable_ms
-
-**Definition Location:** `src/v/serde/rw/chrono.h:20`
-
-**C++ Expression:**
-```cpp
-std::chrono::duration_cast<std::chrono::milliseconds>(
-  std::chrono::nanoseconds::max()
-)
-```
-
-**Calculation:**
-1. `std::chrono::nanoseconds::max()` = `std::numeric_limits<int64_t>::max()` = `9223372036854775807` nanoseconds
-2. Convert to milliseconds (truncating division): `9223372036854775807 / 1000000` = `9223372036854` milliseconds
-3. This is approximately **292.47 years**
-
-**Verification:**
-```python
-import sys
-
-# std::chrono::nanoseconds uses int64_t for rep
-max_int64 = 9223372036854775807
-
-# Convert nanoseconds to milliseconds (duration_cast truncates)
-max_ms = max_int64 // 1000000
-
-print(f'max_serializable_ms = {max_ms} ms')
-print(f'Which is approximately {max_ms / (1000 * 60 * 60 * 24 * 365):.2f} years')
-```
-
-Output:
-```
-max_serializable_ms = 9223372036854 ms
-Which is approximately 292.47 years
-```
-
-## Result
-
-After the fix, the extracted schema correctly shows:
-```json
-{
-  "log_message_timestamp_before_max_ms": {
-    "default": 9223372036854,  // ✅ Correct numeric value
-    "type": "integer"
-  }
-}
-```
-
-## Adding New Computed Constants
-
-To add support for new computed constants:
-
-1. **Find the definition** in the Redpanda source code
-2. **Compute the value** - either manually or with a Python/C++ test
-3. **Add to COMPUTED_CONSTANTS** dictionary in `transformers.py`:
-   ```python
-   COMPUTED_CONSTANTS = {
-       "existing_constant": 12345,
-       "new_constant_name": computed_value,  # Add comment with source location
-   }
-   ```
-4. **Add a test** in `tests/test_known_values.py` to verify resolution
-5. **Document the calculation** with comments including:
-   - Source file location
-   - C++ expression
-   - Calculation steps
-   - Human-readable interpretation
-
-## Test Coverage
-
-All 53 tests pass, including the new test for `max_serializable_ms` resolution:
-
-```bash
-cd tools/property-extractor
-python -m pytest tests/ -v --tb=short
-# ================================================= 53 passed =================================================
-```
-
-## Benefits
-
-✅ **Accurate defaults** - Properties show actual numeric values instead of symbolic names
-✅ **Type safety** - Numeric properties have numeric defaults, not strings
-✅ **Documentation quality** - Users see real values they can use
-✅ **Maintainability** - Centralized mapping makes updates easy
-✅ **Test coverage** - Ensures constants resolve correctly
-
-## Related Files
-
-- `tools/property-extractor/transformers.py` - COMPUTED_CONSTANTS dictionary and resolution logic
-- `tools/property-extractor/tests/test_known_values.py` - Test for constant resolution
-- `src/v/serde/rw/chrono.h` - Source definition of max_serializable_ms
-- `tools/property-extractor/CI_INTEGRATION.md` - Updated test count documentation