profiler-md 0.8.0 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,162 +1,288 @@
1
1
  ---
2
2
  name: profile-optimize
3
3
  description: |
4
- Generate a performance profile, identify hotspots, and optimize.
5
- argument-hint: '[what to focus on]'
4
+ Profile a program, find its costliest functions or nodes, and lower their
5
+ cost.
6
+ argument-hint: '[what to optimize or focus on]'
6
7
  allowed-tools: Bash(profiler-md *)
7
8
  ---
8
9
 
9
- Generate a performance profile, identify hotspots, and optimize.
10
+ Profile a program, find its costliest functions or nodes, and lower their cost.
10
11
 
11
- Focus: $ARGUMENTS
12
+ Arguments: $ARGUMENTS
13
+
14
+ # Goals
15
+
16
+ - Lower the cost named in the arguments, or the cost the user is concerned about
17
+ in the conversation, to the user's target if they set one. A cost is whatever
18
+ the profiler measures: time, memory allocated or retained, instructions, I/O,
19
+ or another metric
20
+ - Keep the program's output unchanged, and keep every test that passed before an
21
+ optimization passing after it. Change the output (e.g. an approximate result)
22
+ only with the user's approval
23
+ - Prove each change with a profile diff
24
+
25
+ If the cost or the code to optimize is unclear from the arguments and the
26
+ conversation, ask the user and stop.
12
27
 
13
28
  # Workflow
14
29
 
15
- 1. Decide what to profile (time or memory) and write an isolated script that
16
- exercises the slow path if nothing runs it directly. Run
17
- `profiler-md --help <language>` for profiling instructions
18
- 2. Capture a baseline: generate a profile, convert it with
19
- `profiler-md path/to/profile`, and read the full report, focusing on:
20
- - Hottest functions: self % shows where time is spent or memory is allocated,
21
- not just passed through
22
- - Hottest call stacks: the call path to the hot functions, with `…` for
23
- frames the entry filter hides
24
- 3. Identify the hottest 1-3 functions by self % and read their source:
25
- - Native functions are often unavoidable but may indicate unnecessary work
26
- (e.g. parsing the same data twice, creating many intermediate arrays)
27
- - Project functions are direct targets
28
- 4. State a hypothesis (see "Hypotheses"). If the bottleneck is unclear, reread
29
- the hot function and its callers
30
- 5. Apply the minimal change that addresses the bottleneck, one optimization at a
31
- time. Leave unrelated code alone. Optimize the general case, not the
32
- benchmark: don't exploit quirks of the profiled input (its size, ordering,
33
- value distribution, or hardcoded special cases for it) unless they're
34
- guaranteed properties of real inputs
35
- 6. Run the relevant tests: if any newly fail, fix the optimization or revert it
36
- 7. Capture a new report as in step 2 and compare self % for the targeted
37
- function(s) against the baseline: if the improvement is negligible or
38
- unclear, revert and go back to step 4 instead of iterating blindly. If the
39
- comparison is too noisy to call, collect more samples (see "Measurement");
40
- don't fall back to manual timing
41
- 8. If the goal is unmet and a clear bottleneck remains, repeat from step 3 with
42
- the new profile as the baseline
43
- 9. Report:
44
- - Before vs after for the hot function(s)
45
- - Any other functions that moved significantly (regressions)
30
+ 1. Pick a profiler for the cost and the program's language. Run
31
+ `profiler-md --help <language>` for each profiler's capture commands and
32
+ settings, and `profiler-md --help` for CLI options
33
+ 2. Write a workload that runs only the code to optimize, unless an existing
34
+ command already does. Keep its input representative of real use. Fix the
35
+ input across runs, because every later profile must exercise the same work
36
+ 3. Capture a baseline profile (e.g. `base.<ext>`), convert it with
37
+ `profiler-md base.<ext>`, and read the whole output
38
+ 4. Read the source of the costliest functions or nodes, and of their callers or
39
+ the nodes that retain them
40
+ - A function or node outside the codebase (e.g. the runtime, a native module,
41
+ or a dependency) cannot be changed, though the code calling it or retaining
42
+ it can. Look for callers that make it repeat work (e.g. parsing the same
43
+ data twice)
44
+ 5. State a hypothesis (see "Hypotheses" and "Focus areas")
45
+ 6. Apply the smallest change that removes the bottleneck, one optimization at a
46
+ time
47
+ - Exploit a property of the workload's input (its size, order, value
48
+ distribution) only when every real input has it
49
+ - Before replacing a representation, an intermediate structure, or a layer
50
+ boundary, follow "Redesigns"
51
+ 7. Run the tests covering the changed code. If one newly fails, fix the change
52
+ or revert it
53
+ 8. Capture a new profile (e.g. `current.<ext>`) with the same workload and
54
+ profiler settings, and diff it with `profiler-md base.<ext> current.<ext>`
55
+ - Keep the change only if the targeted functions' or nodes' cost drops by
56
+ more than their noise floor (see "Measurement"), and no other function's or
57
+ node's cost rises by the amount they dropped. Otherwise revert and return
58
+ to step 5
59
+ - Ignore the profile's total. Its noise floor includes the variation of every
60
+ function or node, so a drop smaller than that floor is indistinguishable
61
+ from noise even when every run repeats the drop
62
+ - If the drop is close to the targeted functions' or nodes' noise floor,
63
+ collect more data (see "Measurement") and diff again
64
+ 9. If the goal is unmet and a clear bottleneck remains, make `current.<ext>` the
65
+ new baseline and return to step 4. If no small change removes the remaining
66
+ cost, follow "Redesigns"
67
+ 10. Once the goal is met, or the remaining cost is spread across functions or
68
+ nodes that no change from "Focus areas" or "Redesigns" reduces, summarize
69
+ the diff between the first baseline and the final profile. List each kept
70
+ optimization and each disproved hypothesis. For a redesign, also report its
71
+ ceiling and the diff from step 4 of "Spike"
46
72
 
47
73
  # Measurement
48
74
 
49
- profiler-md reports are the ONLY measurements in this workflow. NEVER time
50
- manually (e.g. `time` or `performance.now()`).
75
+ - Measure only with profiles converted by `profiler-md`. NEVER time or meter the
76
+ program manually (e.g. `time`, a timer around a call, or the process's memory
77
+ usage). A manual measurement gives one number for the whole run, and that
78
+ number moves with machine load, caches, and garbage collection. A profile diff
79
+ instead attributes each change in cost to functions or nodes, so it shows
80
+ whether the code change removed the targeted cost
81
+ - Two runs of the same workload differ, in the total and in each function or
82
+ node. Before judging the first change, capture a second baseline and diff the
83
+ two. A function's or node's noise floor is how much its cost differs between
84
+ the baselines
85
+ - When a profile has too little data to trust, raise the profiler's resolution
86
+ (e.g. a higher sampling frequency, or a smaller allocation sampling interval),
87
+ enlarge the workload's input, or repeat its work. Apply the same adjustment
88
+ when you recapture the baseline, so both sides of the diff run the same
89
+ workload and settings
51
90
 
52
- Manual timing yields one noisy end-to-end number that can improve for reasons
53
- unrelated to the change (machine load, caches, GC timing), and its
54
- instrumentation distorts the hot path. A profile attributes work to functions,
55
- so a before/after comparison proves the optimization removed the targeted work,
56
- not merely that one run was faster.
91
+ # Hypotheses
57
92
 
58
- If a profile has too few samples to trust, collect more:
93
+ Before changing code, state:
59
94
 
60
- - Run the workload longer or loop it more times
61
- - Raise the sampling frequency if the profiler supports it (see
62
- `profiler-md --help <language>`)
63
- - Regenerate the profile and compare again
95
+ - The bottleneck (e.g. "a new buffer allocated per row", "an `O(n²)` scan for
96
+ duplicates", "the same config parsed on every request")
97
+ - The fix (e.g. "reuse one buffer", "track seen keys in a set", "parse the
98
+ config once at startup")
99
+ - Why the fix lowers the profiled cost, and which functions or nodes should cost
100
+ less in the diff
64
101
 
65
- # Hypotheses
102
+ # Redesigns
103
+
104
+ A local optimization makes the existing design do its work at a lower cost. A
105
+ redesign replaces a representation, an intermediate structure, or a layer
106
+ boundary so the costly work never runs. A redesign changes code the profile
107
+ never ranked and can move cost elsewhere instead of removing it, so test it with
108
+ a spike (see "Spike") before implementing it.
109
+
110
+ ## When to consider one
111
+
112
+ Consider a redesign only when the profile shows any of these signs that the cost
113
+ is in the design rather than in one function or node:
114
+
115
+ - The profile is flat, and many of its costliest functions or nodes implement
116
+ the same abstraction (e.g. each layer converting the same intermediate
117
+ representation)
118
+ - The costliest function or node does only what its interface requires, and
119
+ every local change to it was reverted at step 8
120
+ - The same local fix would repeat at many call sites
121
+ - The design sets the asymptotic bound (e.g. a tree built in full only to be
122
+ walked once)
123
+
124
+ Estimate the redesign's ceiling: the summed cost of the functions or nodes it
125
+ eliminates. If the user set a target and the ceiling falls short of it, report
126
+ that the goal requires a different approach and stop.
66
127
 
67
- Before changing anything, state:
128
+ ## Spike
68
129
 
69
- - What is the bottleneck? (e.g. "repeated object allocation in the hot loop",
70
- "O(n²) suffix scan", "redundant map lookups")
71
- - What is the expected fix? (e.g. "hoist allocation outside loop", "use a
72
- two-pointer suffix scan", "cache the lookup result")
73
- - Why will it be faster?
130
+ A spike is a throwaway implementation of the redesign that covers only the code
131
+ the workload runs. NEVER begin with the redesign itself. Time-box the spike to a
132
+ fraction of the redesign's estimated effort. If the spike overruns, the design
133
+ is too unclear to implement, so discard the spike.
134
+
135
+ 1. State the hypothesis as in "Hypotheses", plus the abstraction that requires
136
+ the work, what replaces the abstraction, the functions or nodes the redesign
137
+ eliminates, and the ceiling
138
+ 2. Write the spike. Hardcode values, skip edge cases and tests, and leave other
139
+ callers on the old code. Isolate it (a copy of the module, its own commit, or
140
+ a separate worktree) so one operation discards it. NEVER edit the existing
141
+ implementation in place
142
+ 3. Check that the spike produces the same program output as the original code on
143
+ the workload
144
+ 4. Capture a profile (e.g. `spike.<ext>`) and diff it against the baseline. The
145
+ eliminated functions or nodes must be gone, and their cost must not have
146
+ moved to other functions or nodes
147
+ 5. If the diff fails either condition in step 4, or the gain falls short of the
148
+ target and no clear change closes the gap, discard the spike and report the
149
+ disproved hypothesis as a finding
150
+ 6. Ask the user to approve the redesign. Present the hypothesis, the ceiling,
151
+ the spike's diff against the baseline, and what the spike skipped. Stop until
152
+ the user answers
153
+ 7. Implement the redesign, then run steps 7 and 8 of the workflow. Also diff the
154
+ new profile against the spike's (`profiler-md spike.<ext> current.<ext>`). A
155
+ function or node that costs more than in the spike shows where the added
156
+ generality reintroduced work. Treat it as a new bottleneck
157
+
158
+ # Commits and pull requests
159
+
160
+ Keep every profile the workflow captures until the commit and pull request
161
+ descriptions include their diffs. Keep the first baseline under its own name, so
162
+ step 9 never overwrites it.
163
+
164
+ - In each commit's description, include the `profiler-md` diff of the profiles
165
+ captured before and after the commit's change (the diff from step 8), in a
166
+ fenced code block. For a redesign, include the diff against the profile from
167
+ before the redesign, not the diff against the spike's profile
168
+ - In a pull request's description, include the `profiler-md` diff between the
169
+ first baseline and the final profile, collapsed in a `<details>` element:
170
+
171
+ ```
172
+ <details>
173
+ <summary>profiler-md diff</summary>
174
+
175
+ <!-- Output of `profiler-md <first baseline> <final profile>` -->
176
+
177
+ </details>
178
+ ```
179
+
180
+ Leave the blank lines around the diff and keep it unindented, so the pull
181
+ request renders it as Markdown
74
182
 
75
183
  # Focus areas
76
184
 
77
- Sub-bullets are examples, not exhaustive lists.
185
+ Each area's sub-bullets are examples.
78
186
 
79
187
  - Time complexity: reduce the asymptotic work per input element
80
- - Replace `O(n²)` scans with single-pass or two-pointer algorithms
81
- - Use heaps for top-N instead of fully sorting
82
- - Sort once and binary search instead of repeated linear scans
188
+ - Replace an `O(n²)` scan with a single pass or two pointers
189
+ - Select the top N with a heap instead of a full sort
190
+ - Sort once and binary search instead of scanning repeatedly
83
191
  - Redundant work: compute each result once
84
- - Avoid parsing or computing the same data twice; cache or memoize repeated
85
- lookups
192
+ - Cache or memoize repeated parses and lookups
86
193
  - Hoist loop-invariant computation out of loops
87
- - Build lookup tables once instead of searching repeatedly; intern repeated
88
- values
194
+ - Build a lookup table once instead of searching repeatedly, and intern
195
+ repeated values
89
196
  - Update incrementally instead of recomputing from scratch
90
- - Persist expensive results between invocations, keyed by input so stale
91
- entries are never served
92
- - Work avoidance: skip or defer computation whose result may never be used
93
- - Exit early once the answer is known
94
- - Filter before expensive transforms, not after
95
- - Order checks cheapest-first so expensive predicates run last and rarely
96
- - Add fast paths for the common input shape
97
- - Defer initialization and large dependency loading until first use,
98
- especially on the startup critical path
99
- - Debounce or throttle work triggered by bursty events
100
- - Data structure selection: match the structure to the access pattern
101
- - Use sets/maps for membership and lookup instead of linear scans
102
- - Use bitsets for dense boolean membership
103
- - Use a ring buffer for FIFO access instead of shifting an array
104
- - Cache locality: lay out data so hot loops read memory sequentially
105
- - Compressed sparse row (CSR) format
106
- - Data-oriented design (struct-of-arrays, packed primitive arrays)
107
- - Sparse arrays indexed by sequential IDs instead of integer-keyed hash maps
197
+ - Skip re-rendering a view whose inputs are unchanged
198
+ - Read all layout values before changing any, so no read forces the layout to
199
+ be recomputed between changes
200
+ - Persist an expensive result between runs, keyed by its input so a stale
201
+ entry is never served
202
+ - Work avoidance: skip or defer work whose result may go unused
203
+ - Return early once the answer is known
204
+ - Filter before an expensive transform
205
+ - Run the cheapest check first, so expensive checks run rarely
206
+ - Add a fast path for the common input shape
207
+ - Process only what the consumer uses (e.g. render only the visible rows of a
208
+ long list)
209
+ - Defer initialization and loading large dependencies until first use,
210
+ especially at startup
211
+ - Debounce or throttle work triggered by bursts of events
212
+ - Data structures: match the structure to the access pattern
213
+ - Look up membership in a set or map instead of a list
214
+ - Use a bitset for dense boolean membership
215
+ - Use a ring buffer for a FIFO queue instead of shifting an array
216
+ - Memory layout: store data compactly and read it sequentially
217
+ - Store graphs in compressed sparse row (CSR) format
218
+ - Use data-oriented design (struct of arrays, packed primitive arrays)
219
+ - Index arrays by sequential IDs instead of hashing integer keys
108
220
  - Iterate in storage order (e.g. row-major for nested arrays)
109
- - Allocation and copying: allocate and copy less in hot paths
110
- - Hoist allocations out of hot loops; reuse buffers instead of reallocating
111
- - Preallocate arrays and buffers to their known final size instead of growing
112
- incrementally
113
- - Avoid intermediate collections from chained transforms; fuse them into one
114
- pass
115
- - Use views/slices over underlying data instead of copying; defer cloning
116
- until mutation is needed
117
- - Memory footprint: hold and retain only what the computation needs
118
- - Stream or process incrementally instead of materializing intermediates
119
- - Bound caches (e.g. LRU); use weak references for object-keyed caches
221
+ - Store each number in the narrowest type that holds its range, pack boolean
222
+ flags into bits, and encode a closed set of values as integers instead of
223
+ strings
224
+ - Allocation and copying: allocate and copy less on the hot path
225
+ - Hoist allocations out of loops, and reuse buffers
226
+ - Preallocate arrays and buffers at their known final size
227
+ - Fuse chained transforms into one pass instead of building a collection per
228
+ step
229
+ - Read through views or slices of the original data, and copy only before a
230
+ mutation
231
+ - Retained memory: keep only what the computation still needs
232
+ - Stream input instead of materializing intermediates
233
+ - Bound caches (e.g. LRU), and key object caches with weak references
120
234
  - Remove listeners, timers, and subscriptions when done
121
- - Avoid closures capturing large outer scopes; drop references held by
122
- long-lived structures once no longer needed
123
- - String handling: minimize the strings created and scanned in hot paths
124
- - Avoid repeated concatenation in loops
125
- - Work with indices or char codes into the original string instead of creating
126
- substrings
127
- - Minimize serialization/deserialization round-trips
128
- - Avoid regex in hot paths
129
- - Runtime friendliness: stay on the JIT's optimized fast paths
130
- - Keep object shapes consistent (same fields, same initialization order)
131
- - Avoid polymorphic call sites and mixed-type arrays in hot paths
132
- - Keep numbers in the runtime's fast representations; avoid deoptimization
133
- triggers in hot loops
134
- - I/O and queries: make fewer round-trips and move fewer bytes
135
- - Read or write in large chunks instead of many small operations
136
- - Eliminate N+1 patterns by coalescing small queries or requests into one
137
- batch
138
- - Push filtering, aggregation, and pagination down to the data store; add
139
- indexes matching the hot query's predicates
140
- - Compress or use binary encodings for large transfers; send only the fields
141
- the consumer reads
142
- - Concurrency: use idle hardware for independent work
143
- - Run independent async operations concurrently, bounded to what the awaited
144
- resource can absorb
145
- - Overlap I/O waits with computation instead of strictly sequencing them
146
- - Split CPU-bound work across threads or processes; use SIMD where the
147
- platform supports it
148
- - Contention and backpressure: keep concurrent parts from waiting on or
149
- overwhelming each other
150
- - Minimize time spent holding locks; shrink critical sections
151
- - Shard or partition shared state; prefer immutable or thread-local data over
152
- shared mutable state
153
- - Bound queues so producers can't outrun consumers; shed or coalesce load when
154
- the system is saturated
155
- - Precision trade-offs: do cheaper work that's good enough, only when the
156
- consumer tolerates the difference
157
- - Use approximate algorithms (sampling, sketches, bloom filters)
158
- - Lower resolution or cap iteration counts when the output tolerance allows
159
- - Observability overhead: keep instrumentation cheap enough that it doesn't
160
- distort the hot path
161
- - Keep logging, tracing, and assertions out of hot loops
162
- - Sample or gate expensive instrumentation behind flags
235
+ - Keep closures from capturing large scopes, and release references a
236
+ long-lived structure no longer needs
237
+ - Strings: create and scan fewer strings on the hot path
238
+ - Build strings with a builder or join instead of repeated concatenation
239
+ - Work with indices into the original string instead of substrings
240
+ - Remove serialization round trips
241
+ - Replace regexes on the hot path with direct character checks
242
+ - Runtime: stay on the runtime's optimized paths
243
+ - Initialize an object's fields in the same order everywhere, so its shape
244
+ stays consistent
245
+ - Keep hot call sites monomorphic and arrays homogeneous
246
+ - Keep numbers in the runtime's unboxed representation, and remove whatever
247
+ deoptimizes a hot loop
248
+ - Per-operation overhead: pay a setup or call cost once for many operations
249
+ instead of once per operation
250
+ - Reuse resources that are expensive to create (connections, threads,
251
+ processes, compiled regexes) through a pool or a long-lived instance
252
+ - Batch calls that cross a boundary (FFI, system calls, IPC, messages to a
253
+ worker) instead of making one call per item
254
+ - Fetch related records in one query or request instead of one per record (the
255
+ N+1 pattern)
256
+ - Read and write in large chunks
257
+ - I/O and queries: move fewer bytes, and push work to the data store
258
+ - Push filtering, aggregation, and pagination down to the data store, and
259
+ index the columns a costly query filters on
260
+ - Compress or binary-encode large transfers, and send only the fields the
261
+ consumer reads
262
+ - Concurrency: run independent work on idle hardware
263
+ - Run independent async operations concurrently, bounded by what the awaited
264
+ resource can serve
265
+ - Overlap I/O waits with computation
266
+ - Split CPU-bound work across threads or processes
267
+ - Contention: keep concurrent work from waiting on or flooding other work
268
+ - Shrink critical sections
269
+ - Shard shared state, or use immutable or thread-local data
270
+ - Bound queues so producers cannot outrun consumers, and shed or coalesce load
271
+ at saturation
272
+ - Specialized implementations: use a faster implementation of the same operation
273
+ - Replace a hand-written routine with the platform's built-in equivalent (e.g.
274
+ its JSON parser or sort)
275
+ - Replace a slow dependency with a faster one that behaves the same
276
+ - Use SIMD, vectorized library calls, or the GPU for data-parallel work
277
+ - Build and runtime configuration: lower the cost without changing the code
278
+ - Build in release mode, with link-time or profile-guided optimization
279
+ - Tune the garbage collector and the heap size to the workload
280
+ - Enable the runtime's JIT or ahead-of-time compilation
281
+ - Precision: compute approximately where the consumer tolerates the error
282
+ - Use sampling, sketches, or Bloom filters
283
+ - Lower resolution or cap iterations within the output's tolerance
284
+ - Instrumentation: keep logging, tracing, and assertions off the hot path
285
+ - Move them out of hot loops
286
+ - Sample them, or gate the expensive ones behind a flag
287
+
288
+ These aren't exhaustive. Reason from first principles when none fits cleanly.