sparkforensics-cli 0.1.0 → 0.2.0

package/vendor-core/docs-content/diagrams/udf-execution-models.dark.svg

@@ -0,0 +1 @@
+<svg id="gd34" width="100%" xmlns="http://www.w3.org/2000/svg" class="flowchart" viewBox="-8 -17.3125 730.3125 506.7875" role="graphics-document document" aria-roledescription="flowchart-v2" style="max-width: 730.3125px;"><style>#gd34{font-family:Recursive,ui-sans-serif,system-ui,-apple-system,sans-serif;font-size:16px;fill:#f4f4f5;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#gd34 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#gd34 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#gd34 .error-icon{fill:#1a232f;}#gd34 .error-text{fill:#e5dcd0;stroke:#e5dcd0;}#gd34 .edge-thickness-normal{stroke-width:1px;}#gd34 .edge-thickness-thick{stroke-width:3.5px;}#gd34 .edge-pattern-solid{stroke-dasharray:0;}#gd34 .edge-thickness-invisible{stroke-width:0;fill:none;}#gd34 .edge-pattern-dashed{stroke-dasharray:3;}#gd34 .edge-pattern-dotted{stroke-dasharray:2;}#gd34 .marker{fill:#a9b5b4;stroke:#a9b5b4;}#gd34 .marker.cross{stroke:#a9b5b4;}#gd34 svg{font-family:Recursive,ui-sans-serif,system-ui,-apple-system,sans-serif;font-size:16px;}#gd34 p{margin:0;}#gd34 .label{font-family:Recursive,ui-sans-serif,system-ui,-apple-system,sans-serif;color:#f4f4f5;}#gd34 .cluster-label text{fill:#f4f4f5;}#gd34 .cluster-label span{color:#f4f4f5;}#gd34 .cluster-label span p{background-color:transparent;}#gd34 .label text,#gd34 span{fill:#f4f4f5;color:#f4f4f5;}#gd34 .node rect,#gd34 .node circle,#gd34 .node ellipse,#gd34 .node polygon,#gd34 .node path{fill:#17202c;stroke:#3a4a52;stroke-width:1px;}#gd34 .rough-node .label text,#gd34 .node .label text,#gd34 .image-shape .label,#gd34 .icon-shape .label{text-anchor:middle;}#gd34 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#gd34 .rough-node .label,#gd34 .node .label,#gd34 .image-shape .label,#gd34 .icon-shape .label{text-align:center;}#gd34 .node.clickable{cursor:pointer;}#gd34 .root .anchor path{fill:#a9b5b4!important;stroke-width:0;stroke:#a9b5b4;}#gd34 .arrowheadPath{fill:rgba(255, 255, 255, 0);}#gd34 .edgePath .path{stroke:#a9b5b4;stroke-width:1px;}#gd34 .flowchart-link{stroke:#a9b5b4;fill:none;}#gd34 .edgeLabel{background-color:#11171a;text-align:center;}#gd34 .edgeLabel p{background-color:#11171a;}#gd34 .edgeLabel rect{opacity:0.5;background-color:#11171a;fill:#11171a;}#gd34 .labelBkg{background-color:rgba(17, 23, 26, 0.5);}#gd34 .cluster rect{fill:#11171a;stroke:#233035;stroke-width:1px;}#gd34 .cluster text{fill:#f4f4f5;}#gd34 .cluster span{color:#f4f4f5;}#gd34 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Recursive,ui-sans-serif,system-ui,-apple-system,sans-serif;font-size:12px;background:#1a232f;border:1px solid hsl(214.2857142857, 0%, 4.3137254902%);border-radius:2px;pointer-events:none;z-index:100;}#gd34 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#f4f4f5;}#gd34 rect.text{fill:none;stroke-width:0;}#gd34 .icon-shape,#gd34 .image-shape{background-color:#11171a;text-align:center;}#gd34 .icon-shape p,#gd34 .image-shape p{background-color:#11171a;padding:2px;}#gd34 .icon-shape .label rect,#gd34 .image-shape .label rect{opacity:0.5;background-color:#11171a;fill:#11171a;}#gd34 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#gd34 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#gd34 .node .neo-node{stroke:#3a4a52;}#gd34 [data-look=&quot;neo&quot;].node rect,#gd34 [data-look=&quot;neo&quot;].cluster rect,#gd34 [data-look=&quot;neo&quot;].node polygon{stroke:url(#gd34-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#gd34 [data-look=&quot;neo&quot;].swimlane.cluster rect{filter:none;}#gd34 [data-look=&quot;neo&quot;].node path{stroke:url(#gd34-gradient);stroke-width:1px;}#gd34 [data-look=&quot;neo&quot;].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#gd34 [data-look=&quot;neo&quot;].node .neo-line path{stroke:#3a4a52;filter:none;}#gd34 [data-look=&quot;neo&quot;].node circle{stroke:url(#gd34-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#gd34 [data-look=&quot;neo&quot;].node circle .state-start{fill:#000000;}#gd34 [data-look=&quot;neo&quot;].icon-shape .icon{fill:url(#gd34-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#gd34 [data-look=&quot;neo&quot;].icon-shape .icon-neo path{stroke:url(#gd34-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#gd34 :root{--mermaid-font-family:Recursive,ui-sans-serif,system-ui,-apple-system,sans-serif;}</style><g><marker id="gd34_flowchart-v2-pointEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowMarkerPath" style="stroke-width:1;stroke-dasharray:1,0"></path></marker><marker id="gd34_flowchart-v2-pointStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="4.5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 5 L 10 10 L 10 0 z" class="arrowMarkerPath" style="stroke-width:1;stroke-dasharray:1,0"></path></marker><marker id="gd34_flowchart-v2-pointEnd-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="11.5" refY="7" markerUnits="userSpaceOnUse" markerWidth="10.5" markerHeight="14" orient="auto"><path d="M 0 0 L 11.5 7 L 0 14 z" class="arrowMarkerPath" style="stroke-width:0;stroke-dasharray:1,0"></path></marker><marker id="gd34_flowchart-v2-pointStart-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="1" refY="7" markerUnits="userSpaceOnUse" markerWidth="11.5" markerHeight="14" orient="auto"><polygon points="0,7 11.5,14 11.5,0" class="arrowMarkerPath" style="stroke-width:0;stroke-dasharray:1,0"></polygon></marker><marker id="gd34_flowchart-v2-circleEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="11" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width:1;stroke-dasharray:1,0"></circle></marker><marker id="gd34_flowchart-v2-circleStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-1" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width:1;stroke-dasharray:1,0"></circle></marker><marker id="gd34_flowchart-v2-circleEnd-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refY="5" refX="12.25" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width:0;stroke-dasharray:1,0"></circle></marker><marker id="gd34_flowchart-v2-circleStart-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-2" refY="5" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width:0;stroke-dasharray:1,0"></circle></marker><marker id="gd34_flowchart-v2-crossEnd" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="12" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width:2;stroke-dasharray:1,0"></path></marker><marker id="gd34_flowchart-v2-crossStart" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="-1" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width:2;stroke-dasharray:1,0"></path></marker><marker id="gd34_flowchart-v2-crossEnd-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="17.7" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width:2.5"></path></marker><marker id="gd34_flowchart-v2-crossStart-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="-3.5" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width:2.5;stroke-dasharray:1,0"></path></marker><g class="root"><g class="clusters"></g><g class="edgePaths"><path d="M361.156,126.625L361.156,130.792C361.156,134.958,361.156,143.292,361.156,151.625C361.156,159.958,361.156,168.292,361.156,172.458L361.156,176.625" id="gd34-L_PLAIN_ARROW_0" class=" edge-thickness-invisible edge-pattern-solid" data-edge="true" data-et="edge" data-id="L_PLAIN_ARROW_0" data-points="W3sieCI6MzYxLjE1NjI1LCJ5IjoxMjYuNjI1fSx7IngiOjM2MS4xNTYyNSwieSI6MTUxLjYyNX0seyJ4IjozNjEuMTU2MjUsInkiOjE3Ni42MjV9XQ==" data-look="classic" style=";"></path><path d="M361.156,295.25L361.156,299.417C361.156,303.583,361.156,311.917,361.156,320.25C361.156,328.583,361.156,336.917,361.156,341.083L361.156,345.25" id="gd34-L_ARROW_PANDAS_0" class=" edge-thickness-invisible edge-pattern-solid" data-edge="true" data-et="edge" data-id="L_ARROW_PANDAS_0" data-points="W3sieCI6MzYxLjE1NjI1LCJ5IjoyOTUuMjV9LHsieCI6MzYxLjE1NjI1LCJ5IjozMjAuMjV9LHsieCI6MzYxLjE1NjI1LCJ5IjozNDUuMjV9XQ==" data-look="classic" style=";"></path></g><g class="edgeLabels"><g class="edgeLabel"><g class="label" data-id="L_PLAIN_ARROW_0" transform="translate(0, -10.4609375)"><text y="-10.1" text-anchor="middle"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em" text-anchor="middle"></tspan></text></g></g><g><rect class="background" style="stroke: none"></rect></g><g class="edgeLabel"><g class="label" data-id="L_ARROW_PANDAS_0" transform="translate(0, -10.4609375)"><text y="-10.1" text-anchor="middle"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em" text-anchor="middle"></tspan></text></g></g><g><rect class="background" style="stroke: none"></rect></g></g><g class="nodes"><g class="root" transform="translate(0, 337.25)"><g class="clusters"><g class="cluster " id="gd34-PANDAS" data-look="classic"><rect x="8" y="8" width="706.3125" height="136.225"></rect><g class="cluster-label " transform="translate(262.21484375, 8)"><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">pandas</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> UDF</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> (vectorized)</tspan></tspan></text></g></g></g></g><g class="edgePaths"><path d="M134.969,76.113L155.96,76.113C176.951,76.113,218.932,76.113,260.247,76.113C301.563,76.113,342.211,76.113,362.535,76.113L382.859,76.113" id="gd34-L_JVM3_PY3_0" class=" edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" data-edge="true" data-et="edge" data-id="L_JVM3_PY3_0" data-points="W3sieCI6MTM0Ljk2ODc1LCJ5Ijo3Ni4xMTI1fSx7IngiOjI2MC45MTQwNjI1LCJ5Ijo3Ni4xMTI1fSx7IngiOjM4Ni44NTkzNzUsInkiOjc2LjExMjV9XQ==" data-look="classic" marker-end="url(#gd34_flowchart-v2-pointEnd)" style=";"></path></g><g class="edgeLabels"><g class="edgeLabel" transform="translate(260.9140625, 76.1125)"><g class="label" data-id="L_JVM3_PY3_0" transform="translate(0, -19.2609375)"><g><rect class="background" x="-88.4453125" y="-0.8515625" width="176.890625" height="40.225"></rect><text y="-10.1" text-anchor="middle"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">whole</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> Arrow</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> batches,</tspan></tspan><tspan class="text-outer-tspan row" x="0" y="1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">no</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> per-row</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> handoff</tspan></tspan></text></g></g></g></g><g class="nodes"><g class="node default  " id="gd34-flowchart-JVM3-4" data-look="classic" transform="translate(90.234375, 76.1125)"><rect class="basic label-container" x="-44.734375" y="-24.3125" width="89.46875" height="48.625"></rect><g class="label" transform="translate(0, -9.3125)"><rect></rect><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">JVM</tspan></tspan></text></g></g></g><g class="node default  " id="gd34-flowchart-PY3-5" data-look="classic" transform="translate(531.8359375, 76.1125)"><rect class="basic label-container" x="-144.9765625" y="-33.1125" width="289.953125" height="66.225"></rect><g class="label" transform="translate(0, -18.1125)"><rect></rect><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">Python</tspan></tspan><tspan class="text-outer-tspan row" x="0" y="1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">(pandas</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> Series</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> /</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> DataFrame)</tspan></tspan></text></g></g></g></g></g><g class="root" transform="translate(99.125, 168.625)"><g class="clusters"><g class="cluster " id="gd34-ARROW" data-look="classic"><rect x="8" y="8" width="508.0625" height="118.625"></rect><g class="cluster-label " transform="translate(146.67578125, 8)"><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">Arrow-optimized</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> Python</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> UDF</tspan></tspan></text></g></g></g></g><g class="edgePaths"><path d="M134.969,67.313L153.98,67.313C172.991,67.313,211.013,67.313,248.368,67.313C285.724,67.313,322.413,67.313,340.757,67.313L359.102,67.313" id="gd34-L_JVM2_PY2_0" class=" edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" data-edge="true" data-et="edge" data-id="L_JVM2_PY2_0" data-points="W3sieCI6MTM0Ljk2ODc1LCJ5Ijo2Ny4zMTI1fSx7IngiOjI0OS4wMzUxNTYyNSwieSI6NjcuMzEyNX0seyJ4IjozNjMuMTAxNTYyNSwieSI6NjcuMzEyNX1d" data-look="classic" marker-end="url(#gd34_flowchart-v2-pointEnd)" style=";"></path></g><g class="edgeLabels"><g class="edgeLabel" transform="translate(249.03515625, 67.3125)"><g class="label" data-id="L_JVM2_PY2_0" transform="translate(0, -19.2609375)"><g><rect class="background" x="-76.56640625" y="-0.8515625" width="153.1328125" height="40.225"></rect><text y="-10.1" text-anchor="middle"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">Arrow</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> transfer,</tspan></tspan><tspan class="text-outer-tspan row" x="0" y="1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">still</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> row</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> semantics</tspan></tspan></text></g></g></g></g><g class="nodes"><g class="node default  " id="gd34-flowchart-JVM2-2" data-look="classic" transform="translate(90.234375, 67.3125)"><rect class="basic label-container" x="-44.734375" y="-24.3125" width="89.46875" height="48.625"></rect><g class="label" transform="translate(0, -9.3125)"><rect></rect><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">JVM</tspan></tspan></text></g></g></g><g class="node default  " id="gd34-flowchart-PY2-3" data-look="classic" transform="translate(420.83203125, 67.3125)"><rect class="basic label-container" x="-57.73046875" y="-24.3125" width="115.4609375" height="48.625"></rect><g class="label" transform="translate(0, -9.3125)"><rect></rect><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">Python</tspan></tspan></text></g></g></g></g></g><g class="root" transform="translate(87.22265625, 0)"><g class="clusters"><g class="cluster " id="gd34-PLAIN" data-look="classic"><rect x="8" y="8" width="531.8671875" height="118.625"></rect><g class="cluster-label " transform="translate(205.2578125, 8)"><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">Plain</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> Python</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> UDF</tspan></tspan></text></g></g></g></g><g class="edgePaths"><path d="M134.969,67.313L155.964,67.313C176.958,67.313,218.948,67.313,260.271,67.313C301.594,67.313,342.25,67.313,362.578,67.313L382.906,67.313" id="gd34-L_JVM1_PY1_0" class=" edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" data-edge="true" data-et="edge" data-id="L_JVM1_PY1_0" data-points="W3sieCI6MTM0Ljk2ODc1LCJ5Ijo2Ny4zMTI1fSx7IngiOjI2MC45Mzc1LCJ5Ijo2Ny4zMTI1fSx7IngiOjM4Ni45MDYyNSwieSI6NjcuMzEyNX1d" data-look="classic" marker-end="url(#gd34_flowchart-v2-pointEnd)" style=";"></path></g><g class="edgeLabels"><g class="edgeLabel" transform="translate(260.9375, 67.3125)"><g class="label" data-id="L_JVM1_PY1_0" transform="translate(0, -28.0609375)"><g><rect class="background" x="-88.46875" y="-0.8515625" width="176.9375" height="57.825"></rect><text y="-10.1" text-anchor="middle"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">pickle,</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> row-at-a-time</tspan></tspan><tspan class="text-outer-tspan row" x="0" y="1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">(serialization</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> cost</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> per</tspan></tspan><tspan class="text-outer-tspan row" x="0" y="2.1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">row)</tspan></tspan></text></g></g></g></g><g class="nodes"><g class="node default  " id="gd34-flowchart-JVM1-0" data-look="classic" transform="translate(90.234375, 67.3125)"><rect class="basic label-container" x="-44.734375" y="-24.3125" width="89.46875" height="48.625"></rect><g class="label" transform="translate(0, -9.3125)"><rect></rect><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">JVM</tspan></tspan></text></g></g></g><g class="node default  " id="gd34-flowchart-PY1-1" data-look="classic" transform="translate(444.63671875, 67.3125)"><rect class="basic label-container" x="-57.73046875" y="-24.3125" width="115.4609375" height="48.625"></rect><g class="label" transform="translate(0, -9.3125)"><rect></rect><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">Python</tspan></tspan></text></g></g></g></g></g></g></g></g><defs><filter id="gd34-drop-shadow" height="130%" width="130%"><feDropShadow dx="4" dy="4" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"></feDropShadow></filter></defs><defs><filter id="gd34-drop-shadow-small" height="150%" width="150%"><feDropShadow dx="2" dy="2" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"></feDropShadow></filter></defs><linearGradient id="gd34-gradient" gradientUnits="objectBoundingBox" x1="0%" y1="0%" x2="100%" y2="0%"><stop offset="0%" stop-color="#3a4a52" stop-opacity="1"></stop><stop offset="100%" stop-color="hsl(200, 0%, 0%)" stop-opacity="1"></stop></linearGradient></svg>

package/vendor-core/docs-content/diagrams/udf-execution-models.svg

@@ -0,0 +1 @@
+<svg id="gd33" width="100%" xmlns="http://www.w3.org/2000/svg" class="flowchart" viewBox="-8 -17.3125 730.3125 506.7875" role="graphics-document document" aria-roledescription="flowchart-v2" style="max-width: 730.3125px;"><style>#gd33{font-family:Recursive,ui-sans-serif,system-ui,-apple-system,sans-serif;font-size:16px;fill:#151b1e;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#gd33 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#gd33 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#gd33 .error-icon{fill:#eef1f6;}#gd33 .error-text{fill:#110e09;stroke:#110e09;}#gd33 .edge-thickness-normal{stroke-width:1px;}#gd33 .edge-thickness-thick{stroke-width:3.5px;}#gd33 .edge-pattern-solid{stroke-dasharray:0;}#gd33 .edge-thickness-invisible{stroke-width:0;fill:none;}#gd33 .edge-pattern-dashed{stroke-dasharray:3;}#gd33 .edge-pattern-dotted{stroke-dasharray:2;}#gd33 .marker{fill:#4f5f60;stroke:#4f5f60;}#gd33 .marker.cross{stroke:#4f5f60;}#gd33 svg{font-family:Recursive,ui-sans-serif,system-ui,-apple-system,sans-serif;font-size:16px;}#gd33 p{margin:0;}#gd33 .label{font-family:Recursive,ui-sans-serif,system-ui,-apple-system,sans-serif;color:#151b1e;}#gd33 .cluster-label text{fill:#151b1e;}#gd33 .cluster-label span{color:#151b1e;}#gd33 .cluster-label span p{background-color:transparent;}#gd33 .label text,#gd33 span{fill:#151b1e;color:#151b1e;}#gd33 .node rect,#gd33 .node circle,#gd33 .node ellipse,#gd33 .node polygon,#gd33 .node path{fill:#ffffff;stroke:#9aa8a3;stroke-width:1px;}#gd33 .rough-node .label text,#gd33 .node .label text,#gd33 .image-shape .label,#gd33 .icon-shape .label{text-anchor:middle;}#gd33 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#gd33 .rough-node .label,#gd33 .node .label,#gd33 .image-shape .label,#gd33 .icon-shape .label{text-align:center;}#gd33 .node.clickable{cursor:pointer;}#gd33 .root .anchor path{fill:#4f5f60!important;stroke-width:0;stroke:#4f5f60;}#gd33 .arrowheadPath{fill:rgba(255, 255, 255, 0);}#gd33 .edgePath .path{stroke:#4f5f60;stroke-width:1px;}#gd33 .flowchart-link{stroke:#4f5f60;fill:none;}#gd33 .edgeLabel{background-color:#eef1f6;text-align:center;}#gd33 .edgeLabel p{background-color:#eef1f6;}#gd33 .edgeLabel rect{opacity:0.5;background-color:#eef1f6;fill:#eef1f6;}#gd33 .labelBkg{background-color:rgba(238, 241, 246, 0.5);}#gd33 .cluster rect{fill:#f2f5f9;stroke:#dbe2df;stroke-width:1px;}#gd33 .cluster text{fill:#151b1e;}#gd33 .cluster span{color:#151b1e;}#gd33 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Recursive,ui-sans-serif,system-ui,-apple-system,sans-serif;font-size:12px;background:#eef1f6;border:1px solid hsl(217.5, 0%, 84.9019607843%);border-radius:2px;pointer-events:none;z-index:100;}#gd33 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#151b1e;}#gd33 rect.text{fill:none;stroke-width:0;}#gd33 .icon-shape,#gd33 .image-shape{background-color:#eef1f6;text-align:center;}#gd33 .icon-shape p,#gd33 .image-shape p{background-color:#eef1f6;padding:2px;}#gd33 .icon-shape .label rect,#gd33 .image-shape .label rect{opacity:0.5;background-color:#eef1f6;fill:#eef1f6;}#gd33 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#gd33 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#gd33 .node .neo-node{stroke:#9aa8a3;}#gd33 [data-look=&quot;neo&quot;].node rect,#gd33 [data-look=&quot;neo&quot;].cluster rect,#gd33 [data-look=&quot;neo&quot;].node polygon{stroke:url(#gd33-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#gd33 [data-look=&quot;neo&quot;].swimlane.cluster rect{filter:none;}#gd33 [data-look=&quot;neo&quot;].node path{stroke:url(#gd33-gradient);stroke-width:1px;}#gd33 [data-look=&quot;neo&quot;].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#gd33 [data-look=&quot;neo&quot;].node .neo-line path{stroke:#9aa8a3;filter:none;}#gd33 [data-look=&quot;neo&quot;].node circle{stroke:url(#gd33-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#gd33 [data-look=&quot;neo&quot;].node circle .state-start{fill:#000000;}#gd33 [data-look=&quot;neo&quot;].icon-shape .icon{fill:url(#gd33-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#gd33 [data-look=&quot;neo&quot;].icon-shape .icon-neo path{stroke:url(#gd33-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#gd33 :root{--mermaid-font-family:Recursive,ui-sans-serif,system-ui,-apple-system,sans-serif;}</style><g><marker id="gd33_flowchart-v2-pointEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowMarkerPath" style="stroke-width:1;stroke-dasharray:1,0"></path></marker><marker id="gd33_flowchart-v2-pointStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="4.5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 5 L 10 10 L 10 0 z" class="arrowMarkerPath" style="stroke-width:1;stroke-dasharray:1,0"></path></marker><marker id="gd33_flowchart-v2-pointEnd-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="11.5" refY="7" markerUnits="userSpaceOnUse" markerWidth="10.5" markerHeight="14" orient="auto"><path d="M 0 0 L 11.5 7 L 0 14 z" class="arrowMarkerPath" style="stroke-width:0;stroke-dasharray:1,0"></path></marker><marker id="gd33_flowchart-v2-pointStart-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="1" refY="7" markerUnits="userSpaceOnUse" markerWidth="11.5" markerHeight="14" orient="auto"><polygon points="0,7 11.5,14 11.5,0" class="arrowMarkerPath" style="stroke-width:0;stroke-dasharray:1,0"></polygon></marker><marker id="gd33_flowchart-v2-circleEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="11" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width:1;stroke-dasharray:1,0"></circle></marker><marker id="gd33_flowchart-v2-circleStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-1" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width:1;stroke-dasharray:1,0"></circle></marker><marker id="gd33_flowchart-v2-circleEnd-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refY="5" refX="12.25" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width:0;stroke-dasharray:1,0"></circle></marker><marker id="gd33_flowchart-v2-circleStart-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-2" refY="5" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width:0;stroke-dasharray:1,0"></circle></marker><marker id="gd33_flowchart-v2-crossEnd" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="12" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width:2;stroke-dasharray:1,0"></path></marker><marker id="gd33_flowchart-v2-crossStart" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="-1" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width:2;stroke-dasharray:1,0"></path></marker><marker id="gd33_flowchart-v2-crossEnd-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="17.7" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width:2.5"></path></marker><marker id="gd33_flowchart-v2-crossStart-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="-3.5" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width:2.5;stroke-dasharray:1,0"></path></marker><g class="root"><g class="clusters"></g><g class="edgePaths"><path d="M361.156,126.625L361.156,130.792C361.156,134.958,361.156,143.292,361.156,151.625C361.156,159.958,361.156,168.292,361.156,172.458L361.156,176.625" id="gd33-L_PLAIN_ARROW_0" class=" edge-thickness-invisible edge-pattern-solid" data-edge="true" data-et="edge" data-id="L_PLAIN_ARROW_0" data-points="W3sieCI6MzYxLjE1NjI1LCJ5IjoxMjYuNjI1fSx7IngiOjM2MS4xNTYyNSwieSI6MTUxLjYyNX0seyJ4IjozNjEuMTU2MjUsInkiOjE3Ni42MjV9XQ==" data-look="classic" style=";"></path><path d="M361.156,295.25L361.156,299.417C361.156,303.583,361.156,311.917,361.156,320.25C361.156,328.583,361.156,336.917,361.156,341.083L361.156,345.25" id="gd33-L_ARROW_PANDAS_0" class=" edge-thickness-invisible edge-pattern-solid" data-edge="true" data-et="edge" data-id="L_ARROW_PANDAS_0" data-points="W3sieCI6MzYxLjE1NjI1LCJ5IjoyOTUuMjV9LHsieCI6MzYxLjE1NjI1LCJ5IjozMjAuMjV9LHsieCI6MzYxLjE1NjI1LCJ5IjozNDUuMjV9XQ==" data-look="classic" style=";"></path></g><g class="edgeLabels"><g class="edgeLabel"><g class="label" data-id="L_PLAIN_ARROW_0" transform="translate(0, -10.4609375)"><text y="-10.1" text-anchor="middle"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em" text-anchor="middle"></tspan></text></g></g><g><rect class="background" style="stroke: none"></rect></g><g class="edgeLabel"><g class="label" data-id="L_ARROW_PANDAS_0" transform="translate(0, -10.4609375)"><text y="-10.1" text-anchor="middle"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em" text-anchor="middle"></tspan></text></g></g><g><rect class="background" style="stroke: none"></rect></g></g><g class="nodes"><g class="root" transform="translate(0, 337.25)"><g class="clusters"><g class="cluster " id="gd33-PANDAS" data-look="classic"><rect x="8" y="8" width="706.3125" height="136.225"></rect><g class="cluster-label " transform="translate(262.21484375, 8)"><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">pandas</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> UDF</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> (vectorized)</tspan></tspan></text></g></g></g></g><g class="edgePaths"><path d="M134.969,76.113L155.96,76.113C176.951,76.113,218.932,76.113,260.247,76.113C301.563,76.113,342.211,76.113,362.535,76.113L382.859,76.113" id="gd33-L_JVM3_PY3_0" class=" edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" data-edge="true" data-et="edge" data-id="L_JVM3_PY3_0" data-points="W3sieCI6MTM0Ljk2ODc1LCJ5Ijo3Ni4xMTI1fSx7IngiOjI2MC45MTQwNjI1LCJ5Ijo3Ni4xMTI1fSx7IngiOjM4Ni44NTkzNzUsInkiOjc2LjExMjV9XQ==" data-look="classic" marker-end="url(#gd33_flowchart-v2-pointEnd)" style=";"></path></g><g class="edgeLabels"><g class="edgeLabel" transform="translate(260.9140625, 76.1125)"><g class="label" data-id="L_JVM3_PY3_0" transform="translate(0, -19.2609375)"><g><rect class="background" x="-88.4453125" y="-0.8515625" width="176.890625" height="40.225"></rect><text y="-10.1" text-anchor="middle"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">whole</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> Arrow</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> batches,</tspan></tspan><tspan class="text-outer-tspan row" x="0" y="1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">no</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> per-row</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> handoff</tspan></tspan></text></g></g></g></g><g class="nodes"><g class="node default  " id="gd33-flowchart-JVM3-4" data-look="classic" transform="translate(90.234375, 76.1125)"><rect class="basic label-container" x="-44.734375" y="-24.3125" width="89.46875" height="48.625"></rect><g class="label" transform="translate(0, -9.3125)"><rect></rect><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">JVM</tspan></tspan></text></g></g></g><g class="node default  " id="gd33-flowchart-PY3-5" data-look="classic" transform="translate(531.8359375, 76.1125)"><rect class="basic label-container" x="-144.9765625" y="-33.1125" width="289.953125" height="66.225"></rect><g class="label" transform="translate(0, -18.1125)"><rect></rect><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">Python</tspan></tspan><tspan class="text-outer-tspan row" x="0" y="1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">(pandas</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> Series</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> /</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> DataFrame)</tspan></tspan></text></g></g></g></g></g><g class="root" transform="translate(99.125, 168.625)"><g class="clusters"><g class="cluster " id="gd33-ARROW" data-look="classic"><rect x="8" y="8" width="508.0625" height="118.625"></rect><g class="cluster-label " transform="translate(146.67578125, 8)"><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">Arrow-optimized</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> Python</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> UDF</tspan></tspan></text></g></g></g></g><g class="edgePaths"><path d="M134.969,67.313L153.98,67.313C172.991,67.313,211.013,67.313,248.368,67.313C285.724,67.313,322.413,67.313,340.757,67.313L359.102,67.313" id="gd33-L_JVM2_PY2_0" class=" edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" data-edge="true" data-et="edge" data-id="L_JVM2_PY2_0" data-points="W3sieCI6MTM0Ljk2ODc1LCJ5Ijo2Ny4zMTI1fSx7IngiOjI0OS4wMzUxNTYyNSwieSI6NjcuMzEyNX0seyJ4IjozNjMuMTAxNTYyNSwieSI6NjcuMzEyNX1d" data-look="classic" marker-end="url(#gd33_flowchart-v2-pointEnd)" style=";"></path></g><g class="edgeLabels"><g class="edgeLabel" transform="translate(249.03515625, 67.3125)"><g class="label" data-id="L_JVM2_PY2_0" transform="translate(0, -19.2609375)"><g><rect class="background" x="-76.56640625" y="-0.8515625" width="153.1328125" height="40.225"></rect><text y="-10.1" text-anchor="middle"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">Arrow</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> transfer,</tspan></tspan><tspan class="text-outer-tspan row" x="0" y="1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">still</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> row</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> semantics</tspan></tspan></text></g></g></g></g><g class="nodes"><g class="node default  " id="gd33-flowchart-JVM2-2" data-look="classic" transform="translate(90.234375, 67.3125)"><rect class="basic label-container" x="-44.734375" y="-24.3125" width="89.46875" height="48.625"></rect><g class="label" transform="translate(0, -9.3125)"><rect></rect><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">JVM</tspan></tspan></text></g></g></g><g class="node default  " id="gd33-flowchart-PY2-3" data-look="classic" transform="translate(420.83203125, 67.3125)"><rect class="basic label-container" x="-57.73046875" y="-24.3125" width="115.4609375" height="48.625"></rect><g class="label" transform="translate(0, -9.3125)"><rect></rect><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">Python</tspan></tspan></text></g></g></g></g></g><g class="root" transform="translate(87.22265625, 0)"><g class="clusters"><g class="cluster " id="gd33-PLAIN" data-look="classic"><rect x="8" y="8" width="531.8671875" height="118.625"></rect><g class="cluster-label " transform="translate(205.2578125, 8)"><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">Plain</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> Python</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> UDF</tspan></tspan></text></g></g></g></g><g class="edgePaths"><path d="M134.969,67.313L155.964,67.313C176.958,67.313,218.948,67.313,260.271,67.313C301.594,67.313,342.25,67.313,362.578,67.313L382.906,67.313" id="gd33-L_JVM1_PY1_0" class=" edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" data-edge="true" data-et="edge" data-id="L_JVM1_PY1_0" data-points="W3sieCI6MTM0Ljk2ODc1LCJ5Ijo2Ny4zMTI1fSx7IngiOjI2MC45Mzc1LCJ5Ijo2Ny4zMTI1fSx7IngiOjM4Ni45MDYyNSwieSI6NjcuMzEyNX1d" data-look="classic" marker-end="url(#gd33_flowchart-v2-pointEnd)" style=";"></path></g><g class="edgeLabels"><g class="edgeLabel" transform="translate(260.9375, 67.3125)"><g class="label" data-id="L_JVM1_PY1_0" transform="translate(0, -28.0609375)"><g><rect class="background" x="-88.46875" y="-0.8515625" width="176.9375" height="57.825"></rect><text y="-10.1" text-anchor="middle"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">pickle,</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> row-at-a-time</tspan></tspan><tspan class="text-outer-tspan row" x="0" y="1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">(serialization</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> cost</tspan><tspan font-style="normal" class="text-inner-tspan" font-weight="normal"> per</tspan></tspan><tspan class="text-outer-tspan row" x="0" y="2.1em" dy="1.1em" text-anchor="middle"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">row)</tspan></tspan></text></g></g></g></g><g class="nodes"><g class="node default  " id="gd33-flowchart-JVM1-0" data-look="classic" transform="translate(90.234375, 67.3125)"><rect class="basic label-container" x="-44.734375" y="-24.3125" width="89.46875" height="48.625"></rect><g class="label" transform="translate(0, -9.3125)"><rect></rect><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">JVM</tspan></tspan></text></g></g></g><g class="node default  " id="gd33-flowchart-PY1-1" data-look="classic" transform="translate(444.63671875, 67.3125)"><rect class="basic label-container" x="-57.73046875" y="-24.3125" width="115.4609375" height="48.625"></rect><g class="label" transform="translate(0, -9.3125)"><rect></rect><g><rect class="background" style="stroke: none"></rect><text y="-10.1"><tspan class="text-outer-tspan row" x="0" y="-0.1em" dy="1.1em"><tspan font-style="normal" class="text-inner-tspan" font-weight="normal">Python</tspan></tspan></text></g></g></g></g></g></g></g></g><defs><filter id="gd33-drop-shadow" height="130%" width="130%"><feDropShadow dx="4" dy="4" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"></feDropShadow></filter></defs><defs><filter id="gd33-drop-shadow-small" height="150%" width="150%"><feDropShadow dx="2" dy="2" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"></feDropShadow></filter></defs><linearGradient id="gd33-gradient" gradientUnits="objectBoundingBox" x1="0%" y1="0%" x2="100%" y2="0%"><stop offset="0%" stop-color="#9aa8a3" stop-opacity="1"></stop><stop offset="100%" stop-color="hsl(214.2857142857, 0%, 86.2745098039%)" stop-opacity="1"></stop></linearGradient></svg>

package/vendor-core/docs-content/tuning/broadcast-sizing.md

@@ -0,0 +1,78 @@
+# Broadcast Sizing
+
+<span class="tag">BROADCASTSIZING</span>
+
+## What it is
+
+A join has two ways to move data. The [shuffle join](#joins) forces all-to-all traffic across the
+cluster; the broadcast join ships one side out to every executor and skips the [shuffle](#shuffle)
+entirely. `spark.sql.autoBroadcastJoinThreshold` is the size gate that picks between them:
+it sets the largest table, in bytes, that Spark will broadcast when planning a join, and it
+defaults to 10 MB.[^1][^2] Set it to `-1` and automatic broadcasting is off, so Spark always
+shuffles.[^1]
+
+<img class="light-only" src="../diagrams/broadcast-vs-shuffle.svg" alt="How the small side estimate against the broadcast threshold picks between broadcasting a copy to every executor for a local join and an all-to-all shuffle join, with the broadcast path's driver collect and executor memory risks noted.">
+<img class="dark-only" src="../diagrams/broadcast-vs-shuffle.dark.svg" alt="How the small side estimate against the broadcast threshold picks between broadcasting a copy to every executor for a local join and an all-to-all shuffle join, with the broadcast path's driver collect and executor memory risks noted.">
+
+Given a join's estimated table sizes and that gate, there are two ways the choice can
+go wrong:
+
+- **underBroadcast**: one side is small enough to broadcast but is being shuffled anyway. The
+  join pays for an all-to-all exchange it could have avoided.
+- **overBroadcast**: a side that is too large is being broadcast, which risks running the
+  driver or the executors out of memory.
+
+## How it's detected
+
+Each join carries an estimated size for the side in question, which is compared against
+the broadcast threshold:
+
+| Finding | Condition | What it means |
+|---|---|---|
+| `underBroadcast` | small side is being shuffled instead of broadcast | a cheap broadcast join was left on the table |
+| `overBroadcast` | an oversized side is being broadcast | the broadcast is a memory hazard |
+
+## Why it matters
+
+Broadcasting the small side sends a full copy of that table to every executor. It pays off
+only when network bandwidth and executor memory are not the bottleneck; there, it is far cheaper
+than the shuffle it replaces.[^3] That is the cost an `underBroadcast` finding is
+pointing at: a small dimension going through an all-to-all exchange when a copy on each
+executor would have run fast.
+
+`overBroadcast` is the opposite mistake, and it fails hard. Pushing a table that is too large
+through the broadcast path breaks in two ways: too-large task result errors and out-of-memory
+failures, both because the oversized table has to be collected on the driver and shipped whole
+to every executor.[^4] The size gate exists for exactly this reason: a copy on every executor is
+harmless for a small table and a memory hazard for a large one.
+
+## How to fix it
+
+For `underBroadcast`, let the small side broadcast. If its estimated size sits above the 10 MB
+default but still fits comfortably in executor memory, raise
+`spark.sql.autoBroadcastJoinThreshold` to cover it, or force the strategy with `broadcast(df)`
+on the small side.[^1][^5]
+
+For `overBroadcast`, keep the large side out of the broadcast path. Lower the threshold so the
+oversized table no longer qualifies, or drop the explicit `broadcast()` hint if one is forcing
+it, and let Spark fall back to a shuffle join.
+
+## Confidence
+
+Validated for both branches. `underBroadcast` and `overBroadcast` rest on the same
+well-documented mechanism: the broadcast threshold and its two failure modes are standard Spark
+behavior, so neither branch is experimental.
+
+## Limitations / false-positive risk
+
+The comparison is only as good as the size estimate behind it. Table sizes come from
+statistics that may be stale or missing, so a join that looks mis-sized can be measuring the wrong number. A
+large broadcast can also be deliberate: an operator who has sized executor memory for it may
+broadcast a table well past the default on purpose.
+
+
+[^1]: [Configuration (Spark)](https://spark.apache.org/docs/latest/configuration.html)
+[^2]: [Performance Tuning (Spark SQL, DataFrames and Datasets Guide)](https://spark.apache.org/docs/latest/sql-performance-tuning.html)
+[^3]: *Learning Spark, 2nd Edition*, Damji, Wenig, Das, Lee, ch. 7
+[^4]: *High Performance Spark, 2nd Edition*, Karau, Polak & Warren, ch. 6
+[^5]: [The Apache Spark Optimization Checklist](https://luminousmen.com/post/the-apache-spark-optimization-checklist)

package/vendor-core/docs-content/tuning/cold-start.md

@@ -0,0 +1,81 @@
+# Cold Start
+<span class="tag">COLD</span>
+
+## What it is
+
+Cold start is the gap between when a Spark application starts and when it actually begins doing
+work: the delay before the [DAGScheduler](#spark-architecture) hands off the first stage for execution. In the event
+log, that moment is marked by `SparkListenerStageSubmitted`, whose payload is a direct
+serialization of a `StageInfo`: the event name plus the `stageInfo` and any properties, with no
+task-level data attached[^1]. That shape matches what the event records. The DAGScheduler
+doesn't pre-schedule the whole DAG upfront (it reacts to stage completions, unlocking child
+stages one at a time), and once it has carved the DAG into stages and handed a `TaskSet` for the
+next stage to the TaskScheduler, only then does the TaskScheduler assign individual tasks to
+executors[^2]. `SparkListenerStageSubmitted` therefore captures the moment the DAGScheduler
+creates/submits that `TaskSet`, before any task has actually been placed on an executor.
+
+## How it's detected
+
+`firstStageSubmittedAt − app.startTime` > 30 s → Warning. `firstStageSubmittedAt` is a synthetic
+field: the timestamp of the first `SparkListenerStageSubmitted` event in the event log.
+
+<img class="light-only" src="../diagrams/cold-start-timeline.svg" alt="A timeline from app.startTime through the gap where executors are acquired and no tasks run to firstStageSubmittedAt, with that gap marked as the cold-start delay.">
+<img class="dark-only" src="../diagrams/cold-start-timeline.dark.svg" alt="A timeline from app.startTime through the gap where executors are acquired and no tasks run to firstStageSubmittedAt, with that gap marked as the cold-start delay.">
+
+## Why it matters
+
+Because the event carries only stage-level metadata and fires ahead of any executor task
+placement[^1][^2], a large gap here reflects time spent before the DAGScheduler could submit
+work at all, not time spent inside tasks. It's a signal about scheduling/startup latency,
+distinct from the per-task metrics that describe work once tasks are actually running on
+executors.
+
+## How to fix it
+
+Dynamic-allocation ramp-up isn't directly tied to `firstStageSubmittedAt` timing, but the
+configuration below governs how many executors an application has available at launch, and how
+reliably that number can grow:
+
+- Size executors deliberately instead of defaulting to one large executor per node. For a
+  10-node, 16-core/64GB cluster targeting ~5 cores per executor, the standard derivation reserves
+  1 core per node for daemons (15 usable cores/node, 150 total), divides by 5 cores/executor
+  (30), and subtracts 1 for the YARN ApplicationMaster: 29 executors × 5 cores × ~18GB[^3].
+  Cloudera's equivalent 6-node walkthrough lands on the same shape: 17 executors × 5 cores × 19G,
+  rather than one fat executor per node[^4].
+- If dynamic allocation is enabled, its ability to add executors at all depends on one of a few
+  supporting mechanisms being turned on. One such mechanism is
+  `spark.dynamicAllocation.shuffleTracking.enabled` (default `true` since Spark 3.0), which
+  "enables shuffle file tracking for executors, which allows dynamic allocation without the need
+  for an external shuffle service," and "will try to keep alive executors that are storing
+  shuffle data for active jobs"[^5]. Without shuffle tracking or an external shuffle service
+  configured, dynamic allocation cannot be enabled at all[^5].
+
+Deliberate executor sizing plus the dynamic-allocation precondition (values from the 16-core/64GB-node derivation above):
+
+```properties
+# ~5 cores per executor instead of one fat executor per node
+spark.executor.cores=5
+spark.executor.memory=18g
+
+# Dynamic allocation, with its required shuffle-tracking precondition (default true since Spark 3.0)
+spark.dynamicAllocation.enabled=true
+spark.dynamicAllocation.shuffleTracking.enabled=true
+```
+
+## Limitations / false-positive risk
+
+Some startup latency is unavoidable: a cluster still has to provision before it can run
+anything. The signal measures the gap before the first stage is submitted, and that gap
+can legitimately vary by cluster manager, so a warning here doesn't always mean something
+is wrong.
+
+
+## Related
+
+- **Executor sizing & dynamic allocation:** [Cluster Tuning](#cluster-config)
+
+[^1]: [JsonProtocol.scala](https://raw.githubusercontent.com/apache/spark/v3.5.0/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala)
+[^2]: [Anatomy of a Spark Application](https://luminousmen.com/post/spark-anatomy-of-spark-application)
+[^3]: [Distribution of Executors, Cores and Memory for a Spark Application](https://raw.githubusercontent.com/spoddutur/spark-notes/master/distribution_of_executors_cores_and_memory_for_spark_application.md)
+[^4]: [How to Tune Your Apache Spark Jobs (Part 2)](https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/)
+[^5]: [Configuration (Spark)](https://spark.apache.org/docs/latest/configuration.html)

package/vendor-core/docs-content/tuning/duplicate-plan-subtree.md

@@ -0,0 +1,45 @@
+# Duplicate Plan Subtree
+
+<span class="tag">DUPPLAN</span>
+
+## What it is
+
+The physical plan is the form Spark actually runs: after optimizing the logical plan, the engine produces a plan that "specifies how the logical plan will execute on the cluster," compiling the query into a series of RDDs and transformations[^1]. A duplicated subtree is the same source read, together with the operators stacked above it, appearing in more than one branch of that plan. When the same logic sits in two different branches of a pipeline, you are "basically signing up to recompute everything from scratch multiple times," because every action on an uncached DataFrame starts back at the source and rebuilds all the intermediate steps[^2].
+
+## How it's detected
+
+Exchanges are the giveaway. A [shuffle](#shuffle) or broadcast is an exchange, and these are the points where Spark's pipelining stops: "Spark operators are often pipelined and executed in parallel processes. However, a shuffle or broadcast exchange breaks this pipeline. We call them materialization points and use the term 'query stages' to denote subsections bounded by these materialization points"[^3]. The signal is the same source-plus-operators subtree feeding more than one branch across those materialization boundaries, which is the shape that gets recomputed unless something intervenes.
+
+<img class="light-only" src="../diagrams/duplicate-plan-subtree.svg" alt="Before and after: the same scan and operators feed two branches and run twice, then a single shared node from exchange reuse or a cache feeds both consumers.">
+<img class="dark-only" src="../diagrams/duplicate-plan-subtree.dark.svg" alt="Before and after: the same scan and operators feed two branches and run twice, then a single shared node from exchange reuse or a cache feeds both consumers.">
+
+| Signal | What it points to |
+|---|---|
+| Identical scan + operators in two or more branches | Same subtree recomputed per branch |
+| A materialization point (shuffle/broadcast) below the repeat | The pipeline breaks and the branch restarts from source |
+
+## Why it matters
+
+Left alone, a repeated subtree is repeated work. Each branch re-reads the source and re-runs everything above it, so a subtree that shows up twice is roughly paid for twice[^2]. That cost lands on exactly the expensive parts of a plan (scans and shuffles) rather than on cheap row-level operators.
+
+## How to fix it
+
+Two mechanisms let a plan reuse an identical subtree instead of rebuilding it.
+
+- Exchange reuse leans on the fact that a shuffle already writes its output to disk. Spark "always executes shuffles by first having the 'source' tasks ... write shuffle files to their local disks," so "running a new job over data that's already been shuffled does not rerun the 'source' side of the shuffle. Because the shuffle files were already written to disk earlier, Spark knows that it can use them to run the later stages of the job, and it need not redo the earlier ones. In the Spark UI and logs, you will see the pre-shuffle stages marked as 'skipped'"[^1]. When the same materialized exchange feeds two consumers, the second reads those existing shuffle files instead of recomputing the subtree beneath it.
+- `cache()` / `persist()` truncate recomputation by materializing the result and keying future lookups on it. Persisting "means materializing an RDD (usually by storing it in memory on the executors) for reuse during the current job"[^2], and in the Structured API the lookup is plan-keyed: "caching is done based on the physical plan. This means that we effectively store the physical plan as our key ... and perform a lookup prior to the execution of a Structured job"[^1]. On a hit, Spark returns the stored data rather than rebuilding intermediate steps from the source[^2].
+
+> **PySpark:** if the repeated subtree is a DataFrame you reuse across branches, `df.cache()` (or `df.persist()`) before the branches split lets both read the materialized result instead of recomputing it.
+
+## Confidence
+
+Medium. Inferring a duplicated subtree from the static plan is a medium-confidence signal that requires runtime validation: whether a repeated subtree actually turns into avoided work depends on runtime facts the plan never shows, such as cache hits, retained shuffle files, and eviction[^2].
+
+## Limitations / false-positive risk
+
+Two subtrees that look identical in the printed plan are not always the same recomputed work. The cache "is tied to the analyzed logical plan, not the final optimized one," so semantically identical queries with different analyzed plans miss the cache and recompute even though "the optimizer will eventually produce the same physical plan"[^2]. [Caching](#caching) is also lazy and often partial: it happens "only as they are accessed," "the only blocks that get cached are the ones Spark was forced to touch," and Spark gives "no built-in visibility into how much of your DataFrame is actually cached, or how much has already been evicted"[^2]. Plan-level inference can therefore over-count, flagging repetition that either never materializes or is only partly reused.
+
+
+[^1]: [Spark: The Definitive Guide](https://www.oreilly.com/library/view/spark-the-definitive/9781491912201/), Chambers & Zaharia, chs. 4, 15, 19
+[^2]: [Explaining the mechanics of Spark caching](https://luminousmen.com/post/explaining-the-mechanics-of-spark-caching)
+[^3]: [Adaptive Query Execution: Speeding Up Spark SQL at Runtime](https://www.databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.html)

package/vendor-core/docs-content/tuning/failures.md

@@ -0,0 +1,124 @@
+# Task Failures
+
+<span class="tag">FAIL</span>
+
+## What it is
+
+A task failure is any task end whose `Reason` field is something other than success. Spark's
+event log serializes this as the formatted class name of whichever `TaskEndReason` instance was
+assigned to that task attempt[^1], and the canonical set of reasons is `Success`, `FetchFailed`,
+`ExceptionFailure`, `TaskResultLost`, `TaskKilled`, `TaskCommitDenied`, `ExecutorLostFailure`,
+`Resubmitted`, and `UnknownReason`[^1]. Each carries its own extra detail: a `FetchFailed` records
+a reduce ID and message; an `ExceptionFailure` carries the exception's class name, description,
+and stack trace, plus accumulator updates (falling back to reading them out of the task's metrics
+object for logs written by Spark 1.x); a `TaskKilled` records a kill reason; a `TaskCommitDenied`
+records the job ID, partition ID, and attempt number; an `ExecutorLostFailure` records whether the
+loss was caused by the application, the executor ID, and a loss reason; `UnknownReason` carries no
+extra fields[^1].
+
+## How it's detected
+
+| failed tasks (share) | Level |
+|---|---|
+| > 5% | Warning |
+| > 20% | Critical |
+
+The task-metrics object attached to a `SparkListenerTaskEnd` event is optional in the schema, so
+it isn't guaranteed to be present for every failure. It tends to be populated for reasons where the
+attempt actually ran far enough to produce metrics (an `ExceptionFailure` or `TaskKilled`, for
+instance) but can be absent for a `Resubmitted` attempt that never completed on its executor at
+all[^1]. Detection has to key off the `Reason` field itself rather than assume metrics will always
+be there to inspect.
+
+## Why it matters
+
+Some reasons point at something worse than a one-off hiccup. An
+`ExecutorLostFailure` can be the knock-on effect of a container or pod killed for exceeding its
+memory grant: enabling off-heap memory without also raising `spark.executor.memoryOverhead` (or
+setting `spark.executor.pyspark.memory` explicitly) lets real usage exceed what YARN or Kubernetes
+allocated, and the executor is killed with a generic container-memory error rather than a
+Spark-level exception[^2]. Losing an executor mid-shuffle can also cascade into failures elsewhere:
+if dynamic allocation removes an executor before a shuffle it wrote data for has completed, that
+shuffle output is gone, and downstream tasks reading it fail and force an unnecessary recompute[^3].
+
+## How to fix it
+
+- Read the `Reason` field before treating all failures the same: an `ExceptionFailure` points at
+  application code, and an `ExecutorLostFailure` usually points at memory sizing or infrastructure.
+- For memory-driven `ExecutorLostFailure`s, size `spark.executor.memoryOverhead` (which defaults
+  to `max(384 MB, 10% of executor memory)`) to actually cover off-heap and PySpark memory, since
+  neither is folded into that default budget automatically[^2].
+- For failures caused by dynamic allocation reclaiming an executor mid-shuffle, enable the external
+  [shuffle service](#shuffle) or `spark.dynamicAllocation.shuffleTracking.enabled` so shuffle output can outlive
+  the executor that wrote it, instead of being lost and recomputed[^3].
+
+Config-only levers for memory-driven `ExecutorLostFailure`s and mid-shuffle executor loss:
+
+```properties
+# Cover off-heap + PySpark memory the default overhead budget does NOT include.
+# Default is max(384m, 10% of executor memory); 2g is an example, size to real off-heap/PySpark use.
+spark.executor.memoryOverhead=2g
+
+# Keep shuffle output alive when dynamic allocation reclaims an executor mid-shuffle
+spark.dynamicAllocation.shuffleTracking.enabled=true
+```
+
+## Failed stage
+
+<span class="tag">SFAIL</span>
+
+A single task failing and a whole stage failing are handled by different parts of the
+scheduler. Task retries sit in the TaskScheduler; the stage-level verdict belongs to the
+[DAGScheduler](#spark-architecture), which decides whether the job as a whole lives or dies. When a stage fails,
+it is not abandoned on the spot: the DAGScheduler resubmits it as a fresh attempt, and the
+job only fails once the stage cannot make progress after repeated attempts[^5]. So the
+difference between "retried and succeeds" and "aborts the job" is whether a later
+attempt finishes before the attempt budget runs out.
+
+<img class="light-only" src="../diagrams/retry-escalation-ladder.svg" alt="The layered budgets a repeated failure climbs through, from task retries to stage resubmission to the executor and application ceilings that abort the job.">
+<img class="dark-only" src="../diagrams/retry-escalation-ladder.dark.svg" alt="The layered budgets a repeated failure climbs through, from task retries to stage resubmission to the executor and application ceilings that abort the job.">
+
+That budget is `spark.stage.maxConsecutiveAttempts`, default 4: the number of consecutive
+stage attempts allowed before the stage is aborted[^4]. Fail, resubmit, and complete within
+those attempts and the job stays alive; keep failing until the limit is spent and the job
+aborts.
+
+A `FetchFailed` is the classic trigger. It counts against that stage attempt limit, which is
+why `spark.stage.ignoreDecommissionFetchFailure` (default `true` since 3.4.0) exists: it keeps
+a fetch failure caused by an executor being decommissioned from counting toward
+`spark.stage.maxConsecutiveAttempts`, which by implication means ordinary fetch failures
+otherwise do count toward the abort budget[^4].
+
+Executor exclusion is the other stage-level lever. The `spark.excludeOnFailure.*` family is
+off by default (`spark.excludeOnFailure.enabled` is `false`) and tracks how many tasks must
+fail on one executor within a stage before that executor is excluded for the stage; it can
+also kill executors excluded on a fetch failure[^4]. If exclusion leaves a TaskSet with no
+schedulable executor left, `spark.scheduler.excludeOnFailure.unschedulableTaskSetTimeout`
+(default 120s) caps how long Spark waits to acquire a new executor before aborting that
+TaskSet[^4].
+
+## Confidence
+
+The failed-task-share thresholds and the stage-failure behaviour both map to
+documented Spark scheduler mechanics; no branch here is experimental.
+
+## Limitations / false-positive risk
+
+A high failed-task share is a blunt signal. It can be dominated by one systemic cause, such
+as a single lost executor whose tasks all fail at once, rather than many independent
+failures pointing at a code or data problem. Transient failures that a later attempt retries
+and succeeds are also not always harmful: task retries reset their count on success, so a
+job can carry a nonzero failure share and still finish correctly. Read the `Reason`
+distribution before treating the share as a defect.
+
+
+## Related
+
+- **Executor sizing:** [Cluster Tuning](#cluster-config)
+- **Memory overhead & off-heap:** [Memory Management](#memory-model)
+
+[^1]: [JsonProtocol.scala](https://raw.githubusercontent.com/apache/spark/v3.5.0/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala)
+[^2]: [Dive Into Spark Memory](https://luminousmen.com/post/dive-into-spark-memory)
+[^3]: [Job Scheduling (Apache Spark Documentation)](https://spark.apache.org/docs/latest/job-scheduling.html)
+[^4]: [Spark Configuration (Apache Spark Documentation)](https://spark.apache.org/docs/latest/configuration.html)
+[^5]: [Anatomy of a Spark Application](https://luminousmen.com/post/spark-anatomy-of-spark-application)

package/vendor-core/docs-content/tuning/gc.md

@@ -0,0 +1,110 @@
+# GC Pressure
+
+<span class="tag">GC</span>
+
+## What it is
+
+GC pressure describes how much of an executor's time goes to JVM garbage collection instead
+of running task code. Spark tracks this directly in its per-task metrics: `jvmGCTime` sits
+alongside `executorRunTime`, the elapsed wall-clock time the executor spent running the task
+(as opposed to `executorCpuTime`, which measures CPU time specifically).[^1] `jvmGCTime` is not
+additive on top of that duration: it's defined as the elapsed time the JVM spent in garbage
+collection *while executing the task*, so GC pauses fall inside the same wall-clock window
+`executorRunTime` already measures, not outside it. Summing the two would double-count the GC
+pauses and overstate how long the task actually ran.[^1]
+
+## How it's detected
+
+Because `jvmGCTime` sits inside `executorRunTime` rather than alongside it, the ratio between
+the two gives a bounded read on how much of a task's wall-clock time went to garbage collection:
+
+| gcPct = jvmGCTime / executorRunTime | Level |
+|---|---|
+| > 10% | Warning |
+| > 20% | Critical |
+
+## Why it matters
+
+As gcPct climbs past these thresholds, a growing share of every task's wall-clock time is
+consumed by garbage collection instead of actual computation, so the executor's useful
+throughput drops even while it appears busy. Two causes show up repeatedly: oversized
+executors, and an on-heap memory split pushed too far toward execution/storage.
+
+Sizing an executor with too many cores relative to its heap is one documented trigger: a
+fat-executor configuration using all 16 cores of a node was observed to not only hurt HDFS
+throughput but also "result in excessive garbage [collection]."[^2]
+
+Pushing `spark.memory.fraction` too far toward its upper end has a similar effect through a
+different path. Its complement, `1 − spark.memory.fraction`, is untracked "User Memory"
+reserved for UDFs, Python/Arrow glue, and native library buffers; Spark doesn't manage this
+region at all, and starving it (which is what raising `spark.memory.fraction` toward
+something like 0.9 does) leads to "GC pressure or random OOMs," with no warning before the
+failure.[^3]
+
+## How to fix it
+
+- Keep cores per executor down rather than packing an entire node's cores onto one JVM (the
+  common ~5-cores-per-executor guideline). That avoids the excessive-garbage-collection
+  failure mode documented for fat-executor configurations.[^2]
+- Don't raise `spark.memory.fraction` past a comfortable range chasing more execution/storage
+  memory: doing so starves the untracked user-memory region and risks the same
+  GC-pressure/OOM failure mode.[^3]
+- Move Tungsten's execution and storage buffers off the JVM heap with
+  `spark.memory.offHeap.enabled=true` (and a positive `spark.memory.offHeap.size`, which is
+  required whenever off-heap is enabled[^4]). Off-heap buffers are invisible to the garbage
+  collector, so fewer and smaller live objects need to be tracked, scanned, and copied on the
+  heap, reducing both the frequency and duration of GC pauses.[^3]
+- Tune or switch the GC collector on large heaps. G1GC is Spark's default collector since 4.0
+  (which defaults to JDK 17), and large executor heaps may need `-XX:G1HeapRegionSize` raised
+  as well.[^5] G1's stop-the-world evacuation and full-GC pauses can still spike under heavy
+  old-generation pressure on large heaps: one documented 88GB-heap benchmark saw a job's pause
+  spike to nearly 100 seconds under default G1 settings, and only tuning
+  `InitiatingHeapOccupancyPercent`, `ConcGCThreads`, and RSet-update parameters brought G1 back
+  ahead of Parallel/CMS on both throughput and latency.[^6] ZGC is a documented alternative
+  built for pause-time control: it performs its expensive work concurrently, without stopping
+  application threads for more than a millisecond, with pause times designed to stay
+  independent of heap size from a few hundred megabytes up to 16TB.[^7]
+
+Move buffers off-heap and give G1 room on large heaps (sizes are examples; set to your workload):
+
+```properties
+# Move Tungsten execution/storage buffers off the JVM heap so the GC has fewer objects to scan
+spark.memory.offHeap.enabled=true
+spark.memory.offHeap.size=2g            # must be > 0 whenever off-heap is enabled; 2g is an example
+
+# G1 is the default collector since Spark 4.0 (JDK 17); raise the region size on large heaps
+spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:G1HeapRegionSize=16m   # 16m is an example
+```
+
+## Confidence
+
+The warning and critical thresholds on gcPct are validated: they read a bounded ratio of
+`jvmGCTime` to `executorRunTime`, both first-class per-task metrics Spark records directly,[^1]
+and crossing 10% or 20% marks a real, measurable share of wall-clock time lost to garbage
+collection.
+
+<span class="tag">EXPERIMENTAL</span> A separate signal treats suspiciously *low* GC time as a
+possible cost-model or noise-floor indicator, and that one is an unsourced heuristic with no
+external basis, so treat it as exploratory rather than a confirmed diagnostic.
+
+## Limitations / false-positive risk
+
+A high gcPct can reflect a transient memory spike (a single skewed task, a brief burst of
+large allocations) rather than chronic pressure, so a threshold crossing on one stage does not
+by itself prove a sustained sizing problem; corroborate against the executor's behavior over
+the whole run. The low-GC signal has no external basis at all, and a near-zero reading is at
+least as likely to mean the workload simply is not allocation-heavy as to mean anything is
+wrong.
+
+## Related
+
+- **Why it happens:** [Memory Management](#memory-model)
+- **Executor sizing:** [Cluster Tuning](#cluster-config)
+
+[^1]: [Monitoring and Instrumentation](https://spark.apache.org/docs/latest/monitoring.html#spark-history-server)
+[^2]: [Distribution of Executors, Cores, and Memory for a Spark Application](https://raw.githubusercontent.com/spoddutur/spark-notes/master/distribution_of_executors_cores_and_memory_for_spark_application.md)
+[^3]: [Diving into Spark Memory Management](https://luminousmen.com/post/dive-into-spark-memory)
+[^4]: [Configuration (Spark)](https://spark.apache.org/docs/latest/configuration.html)
+[^5]: [Spark Tuning Guide](https://spark.apache.org/docs/latest/tuning.html)
+[^6]: [Tuning Java Garbage Collection for Apache Spark Applications](https://www.databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html)
+[^7]: [The Z Garbage Collector (ZGC)](https://wiki.openjdk.org/display/zgc)