sparkforensics-server 0.5.1 → 0.6.0

package/vendor-core/docs-content/detection/chrn.md

@@ -3,6 +3,8 @@
 Executors are stood up and torn down again before they can do useful work:
 re-provisioning churn rather than normal scale-down. Raise
 `spark.dynamicAllocation.executorIdleTimeout`, or widen the
-`minExecutors`/`maxExecutors` bounds to reduce flapping. Self-flags a
+`minExecutors`/`maxExecutors` bounds to reduce flapping. With dynamic
+allocation off (`evidence.dynamicAllocation` is `off`) none of those applies:
+the churn is executor loss or preemption, and `remediation` is empty. Self-flags a
 confidence that scales with how far the short-lived-executor share sits
 past the threshold.

package/vendor-core/docs-content/detection/cold.md

@@ -3,4 +3,5 @@
 The first stage waited more than 30 s for an executor. Keep a warm pool of
 executors, or, with dynamic allocation, raise
 `spark.dynamicAllocation.minExecutors`/`initialExecutors` so the app doesn't
-scale up from zero.
+scale up from zero. With dynamic allocation off (`evidence.dynamicAllocation`
+is `off`) no executor-count property applies and `remediation` is empty.

package/vendor-core/docs-content/detection/mem.md

@@ -1,7 +1,7 @@
 ### `MEM`: Memory utilization {#mem}
 
 Executor memory or core capacity may be over- or under-provisioned: more
-than 50% of available core time ran no task, an executor's heap peaked above
+than 50% of allocated core time ran no task, an executor's heap peaked above
 95% of its allocation, or it stayed below 70%. Some
 detail here needs `spark.eventLog.logStageExecutorMetrics=true` on the run
 being analyzed; without it, per-executor memory usage can't be broken down.

package/vendor-core/docs-content/detection/plan.md

@@ -11,7 +11,14 @@ this tag:
   under 3 MB. Compact upstream output, or coalesce before writing.
 - Under-broadcast: the smaller side of a Sort Merge Join looks well under
   the broadcast threshold; consider a `broadcast()` hint or raising
-  `spark.sql.autoBroadcastJoinThreshold`.
+  `spark.sql.autoBroadcastJoinThreshold`. When the effective threshold
+  (logged, else Spark's 10 MiB) already admits the smaller side
+  (`evidence.broadcastThreshold` is `notLimiting`), the threshold is not what
+  stopped the broadcast, so `remediation` is empty and the advice is a hint or
+  table statistics.
 - Over-broadcast: a broadcast exceeds the 1 GB threshold; check for a
   misapplied broadcast hint or a misconfigured
-  `spark.sql.autoBroadcastJoinThreshold`.
+  `spark.sql.autoBroadcastJoinThreshold`. When the effective threshold is
+  below the broadcast or auto-broadcast is disabled (`evidence.broadcastThreshold`
+  is `notLimiting` or `disabled`), a hint forced it: remove the hint, and
+  `remediation` is empty.

package/vendor-core/docs-content/detection/shfl.md

@@ -1,5 +1,11 @@
 ### `SHFL`: Shuffle I/O {#shfl}
 
 Tasks move a large amount of intermediate data between stages. Raise
-`spark.sql.shuffle.partitions`, or add a broadcast join. Only flagged on
-stages that take at least 0.5% of the run.
+`spark.sql.shuffle.partitions`, or add a broadcast join. The partition-count
+advice follows the effective conf (`evidence.partitions`): `raise` when the
+property limits the stage, `sufficient` when the tasks already read a good
+size each, `aqeCoalesced` when AQE merged the partitions (lower
+`spark.sql.adaptive.advisoryPartitionSizeInBytes`), and `ownPartitioning` when
+the property is already high enough (the stage's own `repartition(n)` or RDD
+parallelism limits it). Only flagged on stages that take at least 0.5% of the
+run.

package/vendor-core/docs-content/detection/skew.md

@@ -1,8 +1,15 @@
 ### `SKEW`: Task skew {#skew}
 
 A small number of tasks take much longer than their peers in the same
-stage. For join-driven skew, enable AQE skew-join handling
-(`spark.sql.adaptive.skewJoin.enabled`); otherwise salt the key or
-repartition on a better key. Flagged when P95 task time (the longest task,
-on a stage with fewer than 20 tasks) exceeds 3x the median and the
-recoverable tail is at least 0.5% of the run.
+stage. The fix depends on what the stage reads, which the finding's
+`evidence.origin` records. A stage that reads a shuffle feeding a join
+(`shuffleJoin`) gets AQE skew-join handling
+(`spark.sql.adaptive.skewJoin.enabled`), unless the run's effective conf
+already has it; otherwise salt the key or repartition on a better key. A
+stage that reads files with uneven sizes (`inputScan`) gets compaction of
+small files or a lower `spark.sql.files.maxPartitionBytes`. Any other stage
+(`other`) gets the salting advice and no conf, as a `code` entry in
+`remediation`; so does a join stage whose skew-join handling is already on. Flagged when P95 task time
+(the longest task, on a stage with fewer than 20 tasks) exceeds 3x the median
+and the recoverable tail is at least 0.5% of the run. The median is the
+textbook one: on an even task count, the mean of the two middle values.

package/vendor-core/docs-content/detection/slow.md

@@ -1,6 +1,10 @@
 ### `SLOW`: Stage slowness {#slow}
 
 A stage ran for 15 minutes or more and no slow host was flagged on it. It
-can appear alongside other findings on the same stage. Often a partition-count problem: raise parallelism via
+can appear alongside other findings on the same stage. On a stage that reads a
+shuffle, often a partition-count problem: raise parallelism via
 `spark.sql.shuffle.partitions` or `spark.default.parallelism`, or check for a
-large per-task data volume driving heavy shuffle and spill.
+large per-task data volume driving heavy shuffle and spill. On a stage that
+reads input files and no shuffle, check input file sizes and lower
+`spark.sql.files.maxPartitionBytes`. `evidence.reads` says which case
+applied (`shuffle`, `input` or `other`).

package/vendor-core/docs-content/detection/spill.md

@@ -4,4 +4,7 @@ Tasks are writing data out of memory, which slows execution. Two spill
 patterns get flagged differently: skew spill, where a few heavy tasks spill
 while most don't (rebalance partitioning), and volume spill, where most
 tasks spill because the data genuinely exceeds available memory (add
-partitions or executor memory). Only flagged on stages that take at least 0.5% of the run.
+partitions or executor memory). `spark.sql.shuffle.partitions` is suggested
+only for a stage that reads a shuffle; `evidence.reads` says what the stage
+reads (`shuffle`, `input` or `other`). Only flagged on stages that take at
+least 0.5% of the run.

package/vendor-core/docs-content/detection/strag.md

@@ -1,6 +1,8 @@
 ### `STRAG`: Straggler tasks {#strag}
 
 A few tasks run much slower than the rest of their stage. Rule out a GC
-pause or a slow shuffle fetch before assuming a hardware issue; if a skewed
-key is the real cause, that's a candidate for AQE's skew-join handling. Only
-flagged on stages that take at least 0.5% of the run.
+pause or a slow shuffle fetch before assuming a hardware issue. If uneven
+data is the cause, the advice is the one a skew finding gives for the same
+stage (`evidence.origin`, see `SKEW`): AQE skew-join handling for a shuffle
+feeding a join, file sizes for a scan, salting otherwise. Only flagged on
+stages that take at least 0.5% of the run.

package/vendor-core/docs-content/detection/tiny.md

@@ -1,5 +1,6 @@
 ### `TINY`: Tiny tasks {#tiny}
 
 Many very short tasks add scheduling overhead out of proportion to the work
-each one does. Repartition to fewer, larger tasks. Only flagged on stages that
-take at least 0.5% of the run.
+each one does. Repartition to fewer, larger tasks; a stage that reads a shuffle
+(`evidence.reads` is `shuffle`) can also lower `spark.sql.shuffle.partitions`.
+Only flagged on stages that take at least 0.5% of the run.

package/vendor-core/docs-content/detection/util.md

@@ -1,4 +1,8 @@
 ### `UTIL`: Low utilization {#util}
 
-Allocated executors sit idle for a large share of the application run.
-Consider a smaller cluster, or enable dynamic allocation.
+Allocated executors sit idle for a large share of the application run: under
+60% of the core time the run allocated (cores times the time each executor was
+alive) ran a task. Consider a smaller cluster. With dynamic allocation off,
+either lower `spark.executor.instances` or enable dynamic allocation; the two are
+alternatives. With it on, lower `spark.dynamicAllocation.maxExecutors`, and
+`spark.dynamicAllocation.minExecutors` too when the run sets it above 0.

package/vendor-core/efficiency-model.js

@@ -1,12 +1,13 @@
 // §5 Efficiency/wastage model. DESIGN SPIKE:
 // decomposes available compute-hours into driver-bound vs executor-bound waste, plus two floors.
-import { computeWallClock } from './wall-clock.js';
+import { stageWindows } from './wall-clock.js';
+import { allocatedCoreMs, allocatedCoreMsWithin } from './allocation.js';
 import { computePeakConcurrentCores } from './core-count.js';
                                                                              
 
 export function computeEfficiencyModel({ app, stages, executorsAdded, executorsRemoved, runAggregates }   
                            
-                               
+                                                                      
                                   
                                     
                                       
@@ -18,23 +19,25 @@ export function computeEfficiencyModel({ app, stages, executorsAdded, executorsR
                           
                                               
   {
-  // Capacity is the peak concurrent core count, the one the utilization and idle-cores detectors
-  // use, so the Unused core time tile matches the verdict's idle figure. Summing every addition
-  // (computeTotalCores) counts a replaced executor's cores alongside its replacement's.
-  // `app ?? {}`: callers tolerate a null app (malformed logs); `app!` would crash on app.resources.
-  const totalCores = computePeakConcurrentCores(app ?? {}, executorsAdded, executorsRemoved);
-  const appDurationMs = (app?.endTime ?? 0) - (app?.startTime ?? 0);
-  const wc = computeWallClock(app, stages                                                               );
+  // Capacity is the run's allocation (cores x time alive, metrics.allocation.coreHours), the one the
+  // utilization and idle-cores detectors use, so the Unused core time tile matches the verdict's
+  // idle figure. Peak concurrent cores times the whole run would count cores the run never held
+  // under dynamic allocation. Null without executor cores: no capacity, so no figure.
+  const input = { app, stages, executors: { added: executorsAdded, removed: executorsRemoved } };
+  const allocatedMs = allocatedCoreMs(input);
+  // Split by where the allocated cores were held, from the same alive intervals: inside the union
+  // of the stages' windows, or outside it (startup, gaps between stages, after the last stage).
+  // Executors that join after startup hold no cores there, so spreading the allocation evenly
+  // over the run would bill the driver for cores that never existed.
+  const allocatedActiveCoreMs = allocatedMs == null ? 0 : allocatedCoreMsWithin(input, stageWindows(stages)) ?? 0;
 
-  const availableComputeHours = totalCores * (appDurationMs / 3600000);
+  const availableComputeHours = (allocatedMs ?? 0) / 3600000;
 
-  // Driver-bound: cores idle during startup + gaps + idle (no active stage).
-  const driverIdleMs = wc.startup + wc.gaps + wc.idle;
-  const driverWasteHours = totalCores * (driverIdleMs / 3600000);
+  // Driver-bound: cores held while no stage was active (startup, gaps, idle).
+  const driverWasteHours = Math.max(0, (allocatedMs ?? 0) - allocatedActiveCoreMs) / 3600000;
 
   // Executor-bound: allocated capacity beyond busy cores during stagesActive. Tasks only run in
   // active windows, so whole-run busyCoreMs already lives inside stagesActive.
-  const allocatedActiveCoreMs = totalCores * wc.stagesActive;
   const busyCoreMs = runAggregates?.busyCoreMs ?? 0;
   const executorWasteHours = Math.max(0, allocatedActiveCoreMs - busyCoreMs) / 3600000;
 
@@ -44,6 +47,11 @@ export function computeEfficiencyModel({ app, stages, executorsAdded, executorsR
   let totalTaskMs = 0;
   const perStage = runAggregates?.perStage ?? {};
   for (const k of Object.keys(perStage)) totalTaskMs += perStage[k].totalTaskDurationSum;
+  // The floor is the least time the work takes on the widest the cluster ever was: peak concurrent
+  // cores (not the allocation), summing every addition (computeTotalCores) would count a replaced
+  // executor's cores alongside its replacement's. `app ?? {}`: callers tolerate a null app
+  // (malformed logs); `app!` would crash on app.resources.
+  const totalCores = computePeakConcurrentCores(app ?? {}, executorsAdded, executorsRemoved);
   const floorZeroSkewMs = totalCores > 0 ? totalTaskMs / totalCores : 0;
 
   let dominantWaste                               = null;

package/vendor-core/evidence-report.js

@@ -180,29 +180,29 @@ export const EVIDENCE_SCHEMA_VERSION         = 5;
 // module (stageShape's totalCores, utilization's unrounded fraction) is left off both, so it never
 // reaches the report; adding, renaming or dropping a key here changes the report contract.
 const EVIDENCE_KEYS = {
-  skew: [],
+  skew: ['origin'],
   stageShape: ['rule'],
-  shuffle: [],
-  partitionSizing: ['rule'],
-  spill: ['spillMagnitude'],
+  shuffle: ['partitions'],
+  partitionSizing: ['rule', 'origin', 'partitions'],
+  spill: ['spillMagnitude', 'reads'],
   gc: ['direction'],
   slowHost: ['variant', 'host', 'hostTaskShare', 'hostMeanMs', 'dimension', 'executorId', 'execMaxValue'],
-  stageSlowness: [],
+  stageSlowness: ['reads'],
   stageFailed: ['variant', 'numTasks', 'memoryBytesSpilled', 'failedTaskDetails'],
   failures: ['failedTasks', 'dominantReason', 'dominantError', 'failureGroups', 'otherFailedTasks'],
-  straggler: ['unit', 'speculativeTasks', 'stragglerCount'],
+  straggler: ['unit', 'speculativeTasks', 'stragglerCount', 'origin'],
   speculationWaste: [],
   retryWaste: ['numTasks', 'memoryBytesSpilled', 'retriedTaskDetails'],
-  tinyTask: [],
+  tinyTask: ['reads'],
   incompleteRun: [],
-  coldStart: [],
+  coldStart: ['dynamicAllocation'],
   utilization: ['cpuUtilizationPct'],
   memoryUtilization: ['variant', 'rule', 'executorId', 'heap', 'dataUnavailable'],
   cacheUtilization: [
     'variant', 'rddId', 'rddName', 'memorySize', 'diskSize', 'numCachedPartitions', 'numPartitions', 'dataUnavailable',
   ],
   coreLocality: ['nonLocalTaskCount'],
-  autoscalingChurn: ['shortLivedExecutorCount'],
+  autoscalingChurn: ['shortLivedExecutorCount', 'dynamicAllocation'],
   cachingOpportunity: ['variant', 'relation', 'format', 'relations', 'operator', 'executionIds', 'totalReadBytes'],
   jobFailureRate: ['failedJobs', 'totalJobs', 'failedTasks', 'totalTasks', 'avgJobDurationMs', 'taskFailureRate'],
   configAudit: ['property'],
@@ -211,8 +211,8 @@ const EVIDENCE_KEYS = {
     'groupIndex',
   ],
   smallFiles: ['executionId', 'stageIds', 'fileCount', 'direction', 'nodeName'],
-  underBroadcast: ['executionId', 'stageIds', 'largerSideBytes'],
-  overBroadcast: ['executionId', 'stageIds'],
+  underBroadcast: ['executionId', 'stageIds', 'largerSideBytes', 'broadcastThreshold'],
+  overBroadcast: ['executionId', 'stageIds', 'broadcastThreshold'],
 }                                                                                    ;
 
 // The other direction: an evidence field EVIDENCE_KEYS doesn't list fails here.

package/vendor-core/finding-presentation.js

@@ -8,7 +8,7 @@
 // detectors.ts is type-only, so it is erased at build time. A detector's scope, order and emits
 // list stay on its DETECTORS entry; renderers get them through detectorInfoByType().
                                                    
-                                                      
+                                                                  
                                                          
 
                                                              
@@ -35,7 +35,14 @@ export const shareLabel = (share        )         => `${Math.round(share * 1e6)
 // a generic line reads the same decision and never recommends a switch the row says is on.
 // A finding with no remediation (older or hand-built data) keeps the property wording.
 function switchAlreadyOn(finding                                 , key        )          {
-  return finding.remediation != null && !finding.remediation.some((r) => r.key === key);
+  return finding.remediation != null && !finding.remediation.some((r) => r.kind === 'conf' && r.key === key);
+}
+
+// The properties an idle-capacity finding lowers once dynamic allocation is on, worded as its row
+// does: the executor cap, and the floor when the row's remediation lowers it too.
+function idleCapacityLowering(f                                 )         {
+  const lowersFloor = f.remediation?.some((r) => r.kind === 'conf' && r.key === 'spark.dynamicAllocation.minExecutors');
+  return ` by lowering spark.dynamicAllocation.maxExecutors${lowersFloor ? ' and spark.dynamicAllocation.minExecutors' : ''}`;
 }
 
 const SKEW_JOIN_KEY = 'spark.sql.adaptive.skewJoin.enabled';
@@ -43,8 +50,10 @@ const SKEW_JOIN_ALREADY_ON = 'AQE skew-join handling is already on, so salt the
 const SKEW_JOIN_AQE_OFF = 'AQE is off, so enable it (spark.sql.adaptive.enabled) for skew-join handling to apply; otherwise salt the key or repartition on a better key.';
 
 // The skew-join generic line, worded per the row's remediation: AQE logged off, switch already on, or neither.
-function skewJoinGeneric(f                                 , unset        )         {
-  if (f.remediation?.some((r) => r.key === 'spark.sql.adaptive.enabled')) return SKEW_JOIN_AQE_OFF;
+function skewJoinGeneric(f                                                      , unset        )         {
+  if (f.origin === 'inputScan') return 'Uneven input files: compact small files or split large ones (lower spark.sql.files.maxPartitionBytes).';
+  if (f.origin === 'other') return 'Work is uneven across tasks: salt the key or repartition on a better key.';
+  if (f.remediation?.some((r) => r.kind === 'conf' && r.key === 'spark.sql.adaptive.enabled')) return SKEW_JOIN_AQE_OFF;
   return switchAlreadyOn(f, SKEW_JOIN_KEY) ? SKEW_JOIN_ALREADY_ON : unset;
 }
 const DYNAMIC_ALLOCATION_KEY = 'spark.dynamicAllocation.enabled';
@@ -120,8 +129,9 @@ export const FINDING_PRESENTATION
     tag: 'TINY',
     thresholdSummary: (t) => `${t.minTasks}+ tasks with a median of ${t.maxP50}ms or less and a P95 of ${t.maxP95}ms or less`,
     actionLabel: () => 'Coalesce small tasks',
-    // The shuffle-vs-no-shuffle fix isn't a Finding field, so one sentence covers both.
-    genericRecommendation: () => 'Scheduler overhead may dominate: lower spark.sql.shuffle.partitions, or coalesce down to fewer, larger tasks.',
+    genericRecommendation: (f) => (f.reads != null && f.reads !== 'shuffle'
+      ? 'Scheduler overhead may dominate: coalesce down to fewer, larger tasks.'
+      : 'Scheduler overhead may dominate: lower spark.sql.shuffle.partitions, or coalesce down to fewer, larger tasks.'),
   },
 
   shuffle: {
@@ -129,7 +139,14 @@ export const FINDING_PRESENTATION
     tag: 'SHFL',
     thresholdSummary: (t) => `stage shuffle read above ${t.minBytes / 1048576} MiB`,
     actionLabel: () => 'Reduce shuffle size',
-    genericRecommendation: () => 'Raise spark.sql.shuffle.partitions, or use a broadcast join for the smaller side.',
+    genericRecommendation: (f) => {
+      switch (f.partitions) {
+        case 'sufficient': return 'The tasks are already a good size, so more partitions will not help: use a broadcast join for the smaller side.';
+        case 'aqeCoalesced': return 'AQE coalesced the shuffle: lower spark.sql.adaptive.advisoryPartitionSizeInBytes, or use a broadcast join for the smaller side.';
+        case 'ownPartitioning': return "spark.sql.shuffle.partitions is already high enough, so raise this stage's own partition count (its repartition(n) or RDD parallelism), or use a broadcast join for the smaller side.";
+        default: return 'Raise spark.sql.shuffle.partitions, or use a broadcast join for the smaller side.';
+      }
+    },
   },
   partitionSizing: {
     name: 'partition sizing',
@@ -146,7 +163,9 @@ export const FINDING_PRESENTATION
     genericRecommendation(f) {
       switch (f.rule) {
         case 'shufflePartitionSkew': return skewJoinGeneric(f, 'For join skew, enable AQE skew-join handling (spark.sql.adaptive.skewJoin.enabled); otherwise salt the key or repartition on a better key.');
-        case 'lowShuffleParallelism': return switchAlreadyOn(f, 'spark.sql.shuffle.partitions')
+        case 'lowShuffleParallelism':
+          if (f.partitions === 'aqeCoalesced') return 'AQE coalesced the shuffle into few tasks: lower spark.sql.adaptive.advisoryPartitionSizeInBytes so each partition is smaller.';
+          return switchAlreadyOn(f, 'spark.sql.shuffle.partitions')
           ? "spark.sql.shuffle.partitions is already high enough, so raise this stage's own partition count (its repartition(n) or RDD parallelism) so each partition is smaller."
           : 'Raise spark.sql.shuffle.partitions so each partition is smaller.';
         case 'maxPartitionTooBig': return 'Repartition to break up the oversized partition before this stage.';
@@ -161,7 +180,9 @@ export const FINDING_PRESENTATION
     thresholdSummary: (t) => `single-task disk spill above ${t.singleTaskDiskGiB} GiB`,
     actionLabel: () => 'Reduce spill',
     // The skew/volume classification isn't a Finding field, so one sentence covers both.
-    genericRecommendation: () => 'If the spill is skew-driven, fix task skew first: adding memory will not help. Otherwise raise spark.sql.shuffle.partitions or increase executor memory.',
+    genericRecommendation: (f) => (f.reads != null && f.reads !== 'shuffle'
+      ? 'If the spill is skew-driven, fix task skew first: adding memory will not help. Otherwise increase executor memory or process less data per task.'
+      : 'If the spill is skew-driven, fix task skew first: adding memory will not help. Otherwise raise spark.sql.shuffle.partitions or increase executor memory.'),
   },
 
   gc: {
@@ -219,14 +240,18 @@ export const FINDING_PRESENTATION
     tag: 'SLOW',
     thresholdSummary: () => 'a stage running far longer than its peers, not attributable to a single slow host',
     actionLabel: () => 'Profile slow stage',
-    genericRecommendation: () => 'Often a partition-count problem: raise parallelism via spark.sql.shuffle.partitions or spark.default.parallelism, or check for a large per-task data volume driving heavy shuffle and spill.',
+    genericRecommendation: (f) => f.reads === 'input'
+      ? 'Often too few or too uneven input partitions: check input file sizes and lower spark.sql.files.maxPartitionBytes, or look for a large per-task data volume driving heavy spill.'
+      : f.reads === 'other'
+        ? 'Check what the stage computes and for a large per-task data volume driving heavy spill.'
+        : 'Often a partition-count problem: raise parallelism via spark.sql.shuffle.partitions or spark.default.parallelism, or check for a large per-task data volume driving heavy shuffle and spill.',
   },
   straggler: {
     name: 'straggling task',
     tag: 'STRAG',
     thresholdSummary: () => 'one or more tasks finishing far after the rest of their stage',
     actionLabel: () => 'Fix stragglers',
-    genericRecommendation: () => 'Rule out a GC pause or a slow shuffle fetch before assuming a hardware issue. If a skewed key is the real cause, that is a candidate for AQE\'s skew-join handling.',
+    genericRecommendation: (f) => `Rule out a GC pause or a slow shuffle fetch before assuming a hardware issue. If uneven data is the cause: ${skewJoinGeneric(f, 'for join-driven skew, enable AQE skew-join handling (spark.sql.adaptive.skewJoin.enabled); otherwise salt the key or repartition on a better key.')}`,
   },
   speculationWaste: {
     name: 'speculation waste',
@@ -240,7 +265,9 @@ export const FINDING_PRESENTATION
     tag: 'COLD',
     thresholdSummary: (t) => `the first stage waiting over ${t.gapSeconds}s for an executor`,
     actionLabel: () => 'Pre-warm cluster',
-    genericRecommendation: () => 'Keep a warm pool of idle executors, or if using dynamic allocation, raise the minimum/initial executor count so it does not scale up from zero.',
+    genericRecommendation: (f) => (f.dynamicAllocation === 'off'
+      ? 'Keep a warm pool of idle executors: dynamic allocation is off, so no executor-count property applies.'
+      : 'Keep a warm pool of idle executors, or if using dynamic allocation, raise the minimum/initial executor count so it does not scale up from zero.'),
   },
 
   memoryUtilization: {
@@ -260,8 +287,8 @@ export const FINDING_PRESENTATION
     genericRecommendation(f) {
       switch (f.variant) {
         case 'idleCores': return switchAlreadyOn(f, DYNAMIC_ALLOCATION_KEY)
-          ? 'Dynamic allocation is already on, so reduce cluster size.'
-          : 'Reduce cluster size or enable dynamic allocation.';
+          ? `Dynamic allocation is already on, so reduce cluster size${idleCapacityLowering(f)}.`
+          : 'Either reduce cluster size (spark.executor.instances) or enable dynamic allocation.';
         case 'wasteModel': return 'Review spark.executor.memory and executor count.';
         case 'memoryBand':
           if (f.dataUnavailable) return undefined;
@@ -278,8 +305,8 @@ export const FINDING_PRESENTATION
     thresholdSummary: (t) => `average executor utilization below ${shareLabel(t.minUtil)}`,
     actionLabel: () => 'Reduce cluster size',
     genericRecommendation: (f) => (switchAlreadyOn(f, DYNAMIC_ALLOCATION_KEY)
-      ? 'Dynamic allocation is already on, so consider reducing cluster size.'
-      : 'Consider reducing cluster size or enabling dynamic allocation.'),
+      ? `Dynamic allocation is already on, so consider reducing cluster size${idleCapacityLowering(f)}.`
+      : 'Consider either reducing cluster size (spark.executor.instances) or enabling dynamic allocation.'),
   },
   coreLocality: {
     name: 'core locality',
@@ -322,7 +349,9 @@ export const FINDING_PRESENTATION
     tag: 'CHRN',
     thresholdSummary: (t) => `over ${shareLabel(t.warningPct)} of executors living under ${t.shortLivedMs / 60000} minutes`,
     actionLabel: () => 'Reduce autoscaling churn',
-    genericRecommendation: () => 'This looks like wasteful re-provisioning rather than normal scale-down: consider raising spark.dynamicAllocation.executorIdleTimeout or widening the minExecutors/maxExecutors bounds to reduce flapping.',
+    genericRecommendation: (f) => (f.dynamicAllocation === 'off'
+      ? 'Dynamic allocation is off, so the churn comes from executors lost or preempted: check the cluster manager\'s preemption and executor-loss logs.'
+      : 'This looks like wasteful re-provisioning rather than normal scale-down: consider raising spark.dynamicAllocation.executorIdleTimeout or widening the minExecutors/maxExecutors bounds to reduce flapping.'),
   },
 
   configAudit: CONFIG_AUDIT_PRESENTATION,
@@ -349,14 +378,18 @@ export const FINDING_PRESENTATION
     tag: 'PLAN',
     thresholdSummary: () => 'a join below the configured size floor that skipped broadcast',
     actionLabel: () => 'Use broadcast join',
-    genericRecommendation: () => 'This could have been a broadcast join: consider a broadcast() hint or raising spark.sql.autoBroadcastJoinThreshold.',
+    genericRecommendation: (f) => (f.broadcastThreshold === 'notLimiting'
+      ? 'The threshold already admits the smaller side, so it is not what stopped the broadcast: consider a broadcast() hint or collecting table statistics.'
+      : 'This could have been a broadcast join: consider a broadcast() hint or raising spark.sql.autoBroadcastJoinThreshold.'),
   },
   overBroadcast: {
     name: 'oversized broadcast join',
     tag: 'PLAN',
     thresholdSummary: (t) => `a broadcast over ${t.overBroadcastBytes / 1073741824} GiB`,
     actionLabel: () => 'Fix oversized broadcast',
-    genericRecommendation: (f) => (switchAlreadyOn(f, 'spark.sql.autoBroadcastJoinThreshold')
+    genericRecommendation: (f) => (f.broadcastThreshold === 'notLimiting'
+      ? 'The configured threshold is below this broadcast, so remove the broadcast() hint that forced it.'
+      : switchAlreadyOn(f, 'spark.sql.autoBroadcastJoinThreshold')
       ? 'Automatic broadcast is already disabled, so remove the broadcast() hint that forced it.'
       : 'Check for a misapplied broadcast hint or a misconfigured spark.sql.autoBroadcastJoinThreshold.'),
   },

package/vendor-core/finding-types.js

@@ -10,16 +10,27 @@
                                                                       
                                                           
 
-/** One concrete change a finding's fix makes, alongside its prose `recommendation`. Only a Spark
- * property the detector already names; `suggested` is null when it computes no value. Sizes carry a
- * Spark unit suffix ("384m"), counts are plain numbers, switches are booleans. */
-                              
+/** One concrete change a finding's fix makes, alongside its prose `recommendation`. Entries are
+ * applied together unless the recommendation words them as alternatives ("either ... or"), as an
+ * idle-capacity finding does with dynamic allocation off: then a consumer applies one. */
+                                                            
+
+/** A Spark property the detector already names; `suggested` is null when it computes no value.
+ * Sizes carry a Spark unit suffix ("384m"), counts are plain numbers, switches are booleans. */
+                                  
                
               
                                              
                                               
  
 
+/** A fix no Spark property makes: the job's code or data has to change. `hint` is the same
+ * wording the finding's `recommendation` gives. */
+                                  
+               
+               
+ 
+
 // The columns every finding carries, whatever its detector.
                                          
               
@@ -79,7 +90,22 @@
 
 // ── Per-stage detectors ─────────────────────────────────────────────────────
 
-                                                
+/** What a skewed stage reads, which decides the fix: 'shuffleJoin' (a shuffle feeding a join, where
+ * AQE skew-join handling applies), 'inputScan' (uneven input files) or 'other'. */
+                                                               
+
+/** What a stage reads, by the dominant side: a shuffle, input files, or neither. */
+                                                       
+
+/** Why a shuffle finding's partition-count advice is or is not given: 'raise' (the property limits
+ * the stage), 'sufficient' (tasks are already a good size), 'aqeCoalesced' (AQE merged the
+ * partitions, so the advisory size is the lever) or 'ownPartitioning' (the property is already high
+ * enough, so the stage's own repartition(n) or RDD parallelism is). */
+                                                                                            
+
+                               
+                      
+ 
                                                                             
 
                                      
@@ -90,16 +116,24 @@
                       
  
 
-                                                   
+                                  
+                                 
+ 
                                                                                      
 
                                           
                                                                                 
+                                                    
+                      
+                                                                                                           
+                                 
  
                                                                                                              
 
                                 
                                                 
+                                                                              
+                     
  
                                                                                
 
@@ -124,7 +158,10 @@
  
                                                                                         
 
-                                                         
+                                        
+                                                                                     
+                     
+ 
                                                                                                        
 
                                       
@@ -151,6 +188,8 @@
                         
                            
                          
+                                                                                     
+                      
  
                                                                                            
 
@@ -167,7 +206,10 @@
                     
  
 
-                                                    
+                                   
+                                                                                  
+                     
+ 
                                                                                         
 
 // ── App-level detectors ─────────────────────────────────────────────────────
@@ -176,7 +218,10 @@
 // valueText is always 'missing': the ApplicationEnd event the log lacks.
                                                                                                     
 
-                                                     
+                                    
+                                                                                                
+                                   
+ 
                                                                                            
 
                                       
@@ -184,11 +229,12 @@
                                    
  
                                                                                                 
-                                                                                             
-                      
+                                                                                            
+                                                                                         
                               
                         
                      
+                          
  
 
                                             
@@ -202,11 +248,10 @@
                             
  
                                                                                                                   
-                                                                                               
-                                                                    
+                                                                                         
+                                                                                         
                             
-                              
-                         
+                                     
                          
                           
  
@@ -231,6 +276,8 @@
 
                                            
                                   
+                                                                                           
+                                   
  
                                                                                                                 
 
@@ -296,13 +343,21 @@
                    
  
                                                                                                               
+
+/** How the effective spark.sql.autoBroadcastJoinThreshold relates to the finding: 'limits' (the
+ * property decided it, or is unknown), 'notLimiting' (already admits, or already below, the
+ * broadcast, so something else decided it) or 'disabled' (-1). */
+                                                                       
 
                                                                      
                           
+                                          
  
                                                                                                                           
 
-                                                        
+                                                                    
+                                          
+ 
                                                                                                                        
 
 // ── Unions ──────────────────────────────────────────────────────────────────