@node-projects/speedscope 2.1.0 → 2.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +47 -6
- package/dist/release/file-format-schema.json +6 -0
- package/dist/release/index.html +1 -1
- package/dist/release/re2js.LICENSE +21 -0
- package/dist/release/release.txt +3 -3
- package/dist/release/speedscope-5DZPH3NW.js +244 -0
- package/docs/IMPORT_FORMATS.md +174 -0
- package/docs/JAVA_THREAD_DUMPS.md +8 -3
- package/docs/JAVA_THREAD_DUMP_ZIP.md +65 -0
- package/docs/LARGE_TRACE_PERFORMANCE.md +72 -0
- package/docs/OTEL_PROFILES.md +117 -0
- package/docs/PERF_IMPORT.md +66 -0
- package/docs/PPROF_IMPORT.md +50 -0
- package/docs/PROFILE_COMPARISON.md +118 -0
- package/docs/PROPERTY_TESTS.md +36 -0
- package/docs/PYTHON_CPROFILE.md +25 -0
- package/docs/RELEASING.md +19 -6
- package/docs/SEARCH_AND_LAYOUT.md +128 -3
- package/docs/SINGLE_FILE.md +55 -0
- package/docs/SOURCE_AND_LABELS.md +116 -1
- package/package.json +14 -2
- package/src/lib/file-format-spec.ts +4 -0
- package/dist/release/speedscope-OUBBGH23.js +0 -191
|
@@ -0,0 +1,174 @@
|
|
|
1
|
+
# Import formats and background processing
|
|
2
|
+
|
|
3
|
+
The [README](../README.md#supported-file-formats) lists supported sources.
|
|
4
|
+
This guide describes additional fork-specific behavior and its limits.
|
|
5
|
+
|
|
6
|
+
## Perf clock weights
|
|
7
|
+
|
|
8
|
+
Explicit `perf script` periods now determine aggregate weights. CPU/task-clock
|
|
9
|
+
periods are nanoseconds of sampled CPU time; off-CPU periods remain separate
|
|
10
|
+
sleep-time profiles, and cycles remain raw counts. Missing periods retain
|
|
11
|
+
explicitly labeled inference or sample counts. Timestamp gaps are not claimed
|
|
12
|
+
as measured execution. See [perf semantics and the public regression](PERF_IMPORT.md).
|
|
13
|
+
|
|
14
|
+
## OpenTelemetry Profiles
|
|
15
|
+
|
|
16
|
+
Current OTLP Profiles JSON imports sampled stacks through a shared dictionary,
|
|
17
|
+
preserving metric weights, inline chains and thread/resource separation.
|
|
18
|
+
Timestamps do not become durations, and sample periods do not multiply values.
|
|
19
|
+
The importer currently parses the JSON in memory and creates aggregate profiles;
|
|
20
|
+
binary protobuf and older development layouts remain unsupported. See the
|
|
21
|
+
[pinned schema, limits and exact supported scope](OTEL_PROFILES.md).
|
|
22
|
+
|
|
23
|
+
## Classic Android method traces
|
|
24
|
+
|
|
25
|
+
Open the classic concatenated key/data format emitted by Android method tracing:
|
|
26
|
+
a text key section followed by a little-endian `SLOW` binary section. Versions
|
|
27
|
+
1, 2 and 3 are supported, including version-3 record padding. The file can be
|
|
28
|
+
named `.trace` or renamed; detection uses the format signature.
|
|
29
|
+
|
|
30
|
+
The importer preserves method signatures, thread identities and available Java
|
|
31
|
+
source locations. Clock metadata is required. `thread-cpu` and `wall` clocks
|
|
32
|
+
remain distinct; version-3 `dual` input creates separate labeled CPU and wall
|
|
33
|
+
profiles for each thread. Version-1 `global` means wall time. All values use
|
|
34
|
+
microseconds. CPU clocks are local to their thread; wall offsets remain relative
|
|
35
|
+
to the trace origin, so leading unobserved time can appear as idle weight.
|
|
36
|
+
The absolute date in the binary header is not added to those offsets.
|
|
37
|
+
|
|
38
|
+
Initial method exits reconstruct calls already active at trace start. Calls
|
|
39
|
+
still open at EOF close at that thread's last observed event. These are clipped
|
|
40
|
+
observation windows and do not invent work before or after the recording.
|
|
41
|
+
|
|
42
|
+
Fixed records are scanned without retaining a second parsed-record array.
|
|
43
|
+
Temporary state includes key dictionaries and per-thread stacks, alongside the
|
|
44
|
+
output builders. Key text is decoded line by line with a 16 MiB section limit
|
|
45
|
+
and 1 MiB line limit. The input buffer and final profiles still occupy memory.
|
|
46
|
+
|
|
47
|
+
Raw binary-only/streaming traces, newer ART packet formats, Perfetto protobuf,
|
|
48
|
+
Android systrace HTML and wrapped timestamp recovery are unsupported. Unknown
|
|
49
|
+
clocks, partial records, invalid method actions and decreasing per-thread clocks
|
|
50
|
+
produce explicit errors. See the
|
|
51
|
+
[synthetic fixture, generation script and AOSP references](../sample/profiles/android-method-trace/README.md)
|
|
52
|
+
for the exact supported layouts and clock policy.
|
|
53
|
+
|
|
54
|
+
## Java thread-dump archives
|
|
55
|
+
|
|
56
|
+
Traditional HotSpot `jstack` / `Thread.print` text can be opened directly or as
|
|
57
|
+
an archive of captures from one JVM. Observed stacks contribute raw samples;
|
|
58
|
+
they do not measure elapsed time. The ZIP importer checks declared sizes and
|
|
59
|
+
actual expansion, feeds bounded UTF-8 lines into the parser and supports
|
|
60
|
+
cancellation without returning partial results.
|
|
61
|
+
|
|
62
|
+
See [text snapshot semantics](JAVA_THREAD_DUMPS.md) and the
|
|
63
|
+
[ZIP format and resource policy](JAVA_THREAD_DUMP_ZIP.md). Archive support is
|
|
64
|
+
limited to classic ZIP32 STORE/DEFLATE; it is not a generic archive extractor.
|
|
65
|
+
|
|
66
|
+
## pprof metric and source identity
|
|
67
|
+
|
|
68
|
+
pprof's `default_sample_type` selects a sample metric by its string-table
|
|
69
|
+
identity. Protobuf integer objects are compared by value. If the preferred
|
|
70
|
+
metric is absent or cannot be found, the existing last-metric fallback applies.
|
|
71
|
+
For example, a recording can prefer allocation counts even when a later sample
|
|
72
|
+
column measures bytes; the displayed unit follows the chosen metric.
|
|
73
|
+
|
|
74
|
+
Function IDs and location IDs retain their complete unsigned 64-bit identity.
|
|
75
|
+
They are not truncated to their low 32 bits or rounded through a JavaScript
|
|
76
|
+
number, so IDs that share a low word remain distinct. String-table positions
|
|
77
|
+
must fit safe nonnegative integers. Source lines must fit safe positive integers.
|
|
78
|
+
|
|
79
|
+
Each location retains its own source line, falling back to the function's
|
|
80
|
+
definition line when needed. One location no longer overwrites the shared
|
|
81
|
+
function metadata used by another location. The current importer retains the
|
|
82
|
+
outermost entry of an inline-location chain; expanding all inlined functions
|
|
83
|
+
is outside this change.
|
|
84
|
+
|
|
85
|
+
Negative Go heap delta weights now import as separate negative-magnitude and
|
|
86
|
+
positive-contribution profiles. Compare positive minus negative in absolute mode
|
|
87
|
+
to inspect net changes; see [signed pprof semantics and limits](PPROF_IMPORT.md).
|
|
88
|
+
This does not add arbitrary-precision weights, a metric chooser or full inline
|
|
89
|
+
chains. See the [pprof schema used by the importer](../src/import/profile.proto).
|
|
90
|
+
|
|
91
|
+
## Background import for large recordings
|
|
92
|
+
|
|
93
|
+
### Streaming perf and folded stacks
|
|
94
|
+
|
|
95
|
+
UTF-8 folded stacks and `perf script` text now stream plain or gzip input into
|
|
96
|
+
the same incremental parsers used by ordinary imports. Supported suffixes are
|
|
97
|
+
`.folded`, `.collapsed`, `.stacks`, `.collapsedstack.txt`, `.perf`, `.perf.txt`
|
|
98
|
+
and `.linux-perf.txt`, optionally followed by `.gz`. Ordinary `.txt` files can
|
|
99
|
+
be detected as either format. Other names retain generic format detection.
|
|
100
|
+
|
|
101
|
+
The reader does not retain the complete expanded byte buffer, decoded text or
|
|
102
|
+
an array of all lines. It retains one bounded line and, for perf, one bounded
|
|
103
|
+
event. Input limits are 1 GiB expanded bytes, 1 Mi UTF-16 code units per line,
|
|
104
|
+
4 Mi code units per perf event, 65,536 stack frames and two million samples.
|
|
105
|
+
A limit failure stops import; it does not retry through an unbounded reader.
|
|
106
|
+
Malformed weights and existing perf ordering checks remain enforced.
|
|
107
|
+
|
|
108
|
+
This path also runs for small compressed inputs and when workers are unavailable.
|
|
109
|
+
It yields during parsing, exposes progress and supports cancellation. Gzip needs
|
|
110
|
+
native `DecompressionStream`; unavailable native decompression, UTF-16 input and
|
|
111
|
+
other recognized formats retain the generic fallback. Final trees, sample lists,
|
|
112
|
+
symbol processing and worker snapshots still scale with the recording.
|
|
113
|
+
|
|
114
|
+
### Worker routing and fallback
|
|
115
|
+
|
|
116
|
+
Inputs of at least **1 MiB** use a Web Worker when available, regardless of
|
|
117
|
+
filename extension. This includes perf, folded stacks, pprof, JFR, Android
|
|
118
|
+
method traces, supported ZIP archives and renamed captures. Format detection,
|
|
119
|
+
generic parsing, demangling and profile snapshot preparation happen away from
|
|
120
|
+
the main UI thread. Cancelling the task terminates its worker.
|
|
121
|
+
|
|
122
|
+
Recognized gzip perf/folded/text suffixes prefer a worker at any compressed
|
|
123
|
+
size: a tiny compressed file can expand into a large recording. Small plain
|
|
124
|
+
text stays on the main thread. The bounded text reader yields for cancellation
|
|
125
|
+
when a worker is blocked or unavailable; native gzip streaming requires
|
|
126
|
+
`DecompressionStream`. Unsupported encodings and unavailable streaming APIs use
|
|
127
|
+
the existing generic fallback, which can materialize the full input.
|
|
128
|
+
|
|
129
|
+
The worker first attempts the duration-trace and supported text streaming paths,
|
|
130
|
+
then uses the generic importer when those formats do not apply. Generic fallback
|
|
131
|
+
formats still read/materialize their input and retain their final model; routing them
|
|
132
|
+
through a worker does not make them streaming or out-of-core. Snapshot transfer,
|
|
133
|
+
geometry and final view construction still have memory costs proportional to
|
|
134
|
+
the recording. [Large-trace guidance](LARGE_TRACE_PERFORMANCE.md) describes the
|
|
135
|
+
duration-trace limits and measurements separately.
|
|
136
|
+
|
|
137
|
+
If workers are unavailable or blocked by a host's CSP, the app falls back to
|
|
138
|
+
the main-thread importer. DOM-dependent formats can also fall back after worker
|
|
139
|
+
detection. A transferred input buffer is returned before this fallback, so the
|
|
140
|
+
main thread does not attempt to parse an empty detached buffer. Synchronous
|
|
141
|
+
generic fallback parsing can block the UI until it returns; the streaming text
|
|
142
|
+
and asynchronous ZIP importers yield and check cancellation on the main thread.
|
|
143
|
+
|
|
144
|
+
Oversized supported ZIP files receive a four-byte signature preflight before
|
|
145
|
+
their complete body is read, both in the main importer and worker path. Other
|
|
146
|
+
formats retain their existing input/resource policies; the ZIP compressed-size
|
|
147
|
+
limit is not a blanket cap on every supported recording.
|
|
148
|
+
|
|
149
|
+
### Text-stream measurements
|
|
150
|
+
|
|
151
|
+
Three fresh Node 24.19.0 ARM64 processes per mode imported a synthetic file
|
|
152
|
+
containing one million copies of `root;worker;leaf 1`: 19,000,000 plain bytes
|
|
153
|
+
or 46,140 gzip bytes. All modes produced one profile with total weight 1,000,000.
|
|
154
|
+
The final model is deliberately tiny, which isolates temporary input costs;
|
|
155
|
+
these figures do not predict memory for a million distinct stacks.
|
|
156
|
+
|
|
157
|
+
| Mode | Plain median time / peak RSS | Gzip median time / peak RSS |
|
|
158
|
+
| --- | ---: | ---: |
|
|
159
|
+
| Generic materializing importer | 466 ms / 110 MiB | 853 ms / 300 MiB |
|
|
160
|
+
| Streaming with worker-style scheduling | 573 ms / 86 MiB | 629 ms / 66 MiB |
|
|
161
|
+
| Streaming with main-thread cooperative yields | 993 ms / 85 MiB | 1,092 ms / 66 MiB |
|
|
162
|
+
|
|
163
|
+
Peak RSS columns are medians of each fresh process's peak, not retained model
|
|
164
|
+
heap. Worker-style measurements skip unnecessary timer yields, as the worker
|
|
165
|
+
can be terminated externally. They time the parsing function over a Node file
|
|
166
|
+
stream, excluding actual worker startup, demangling, snapshot transfer and view
|
|
167
|
+
construction. The cooperative path permits UI cancellation when no worker is
|
|
168
|
+
available; Windows timer overhead makes it slower in this test. Plain-text
|
|
169
|
+
streaming trades some throughput for lower memory even without those yields.
|
|
170
|
+
|
|
171
|
+
Reproduce with `node scripts/benchmark-profile.mjs <path> import`,
|
|
172
|
+
`text-stream-import` (worker-style scheduling), or `text-stream-main-import`
|
|
173
|
+
(cooperative scheduling). Do not compare these folded-stack figures with PLC
|
|
174
|
+
duration-trace timings as if they used the same input or parser.
|
|
@@ -3,7 +3,9 @@
|
|
|
3
3
|
Import traditional HotSpot text output from `jstack PID` or
|
|
4
4
|
`jcmd PID Thread.print` by opening a `.jstack` file or a text file containing
|
|
5
5
|
the `Full thread dump …` header. Concatenate captures from the same JVM to
|
|
6
|
-
compare observed stacks across repeated snapshots.
|
|
6
|
+
compare observed stacks across repeated snapshots. You can also open a ZIP of
|
|
7
|
+
traditional text captures from the same JVM; see [archive limits and supported
|
|
8
|
+
ZIP formats](JAVA_THREAD_DUMP_ZIP.md).
|
|
7
9
|
|
|
8
10
|
Each observed Java stack contributes one raw sample. The importer creates an
|
|
9
11
|
aggregate profile per thread name and Java thread ID, falling back to `tid`
|
|
@@ -23,8 +25,11 @@ The parser iterates lines and retains the current stack and current snapshot's
|
|
|
23
25
|
thread identities, rather than collecting every line or observation. The text
|
|
24
26
|
input and resulting profile model still occupy memory; this format does not use
|
|
25
27
|
the duration-trace streaming importer. Prefer captures from one JVM per file.
|
|
26
|
-
ZIP
|
|
27
|
-
|
|
28
|
+
The ZIP importer instead feeds bounded UTF-8 lines into the same parser and
|
|
29
|
+
checks both declared and actual decompressed sizes. Structured virtual-thread
|
|
30
|
+
JSON, native address stacks and lock/state visualization remain outside the
|
|
31
|
+
supported format. Unsupported ZIP variants can be extracted before importing
|
|
32
|
+
their text captures.
|
|
28
33
|
|
|
29
34
|
The checked-in [synthetic example](../sample/profiles/java-thread-dump/synthetic.jstack)
|
|
30
35
|
contains repeated snapshots and source locations. Format references are
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Java thread-dump ZIP archives
|
|
2
|
+
|
|
3
|
+
Open a ZIP containing traditional HotSpot `jstack` / `jcmd Thread.print` text
|
|
4
|
+
captures from **one JVM**. The same thread name and Java thread ID identify a
|
|
5
|
+
thread across snapshots; renamed threads remain separate. Each observed Java
|
|
6
|
+
stack contributes one raw sample, not measured elapsed time. Missing threads
|
|
7
|
+
and entries without Java stacks contribute no samples.
|
|
8
|
+
|
|
9
|
+
Entries are processed in deterministic, case-sensitive filename order, not ZIP
|
|
10
|
+
insertion order or file modification time. For example, `1.jstack`, `10.jstack`,
|
|
11
|
+
then `2.jstack` is the lexical order. The result is aggregate, so this ordering
|
|
12
|
+
does not claim a timeline or attach elapsed durations to snapshot intervals.
|
|
13
|
+
Every `.jstack` entry must contain its own `Full thread dump` header near the
|
|
14
|
+
start. Each entry boundary closes the previous snapshot.
|
|
15
|
+
|
|
16
|
+
The importer recognizes `.jstack`, `.txt`, `.log`, `.dump` and `.tdump` entries
|
|
17
|
+
with a `Full thread dump` header within the first 64 KiB. Other regular files
|
|
18
|
+
and ordinary notes are ignored as profile input, but still undergo bounded
|
|
19
|
+
decompression and checksum validation. Directories are skipped. Text is strict
|
|
20
|
+
UTF-8; UTF-16 dumps should be converted or imported as separate text files.
|
|
21
|
+
|
|
22
|
+
## Resource policy
|
|
23
|
+
|
|
24
|
+
| Resource | Maximum |
|
|
25
|
+
| --- | ---: |
|
|
26
|
+
| Compressed archive | 32 MiB |
|
|
27
|
+
| Archive entries, including directories | 512 |
|
|
28
|
+
| Decompressed bytes per entry | 16 MiB |
|
|
29
|
+
| Total decompressed bytes, including ignored files | 64 MiB |
|
|
30
|
+
| UTF-8 filename bytes | 512 |
|
|
31
|
+
| Path depth | 8 components |
|
|
32
|
+
| Decoded line length | 65,536 UTF-16 code units |
|
|
33
|
+
|
|
34
|
+
Declared sizes are checked before decompression. Actual expanded bytes are
|
|
35
|
+
checked on every output chunk before decoding or parsing, so lying ZIP headers
|
|
36
|
+
cannot bypass the limits. The inflater emits chunks of at most 16 KiB; a limit
|
|
37
|
+
violation can allocate that one output chunk before it is rejected. It never
|
|
38
|
+
accumulates the whole expanded entry. The line decoder retains at most one
|
|
39
|
+
bounded line, while the parser retains the current stack, current snapshot's
|
|
40
|
+
thread identities and the resulting profile builders. The compressed input and
|
|
41
|
+
final profile model still occupy memory; this is not an out-of-core viewer.
|
|
42
|
+
The importer yields during work and supports cancellation without returning a
|
|
43
|
+
partially imported group.
|
|
44
|
+
|
|
45
|
+
## Supported ZIP subset
|
|
46
|
+
|
|
47
|
+
Single-disk ZIP32 archives using **STORE** or **DEFLATE** are supported, including
|
|
48
|
+
ordinary data descriptors. Local/central headers, entry extents, decompressed
|
|
49
|
+
sizes and CRC32 checksums must agree. Paths are validated without extracting
|
|
50
|
+
anything to disk. Absolute/traversal paths, duplicate or case-colliding names,
|
|
51
|
+
file/directory conflicts, symlinks, special files and non-normalized Unicode
|
|
52
|
+
paths are rejected. Names must be ASCII or explicitly UTF-8.
|
|
53
|
+
|
|
54
|
+
Encrypted archives, ZIP64, split archives, self-extracting prefixes, archive
|
|
55
|
+
padding, alternate Unicode-path extra fields and other compression methods are
|
|
56
|
+
unsupported. Extract an unsupported archive yourself and open its text captures
|
|
57
|
+
individually or concatenate them into one `.jstack` file.
|
|
58
|
+
|
|
59
|
+
The checked-in [synthetic archive](../sample/profiles/java-thread-dump/synthetic.zip)
|
|
60
|
+
contains two short snapshots of the same thread. This is an independently
|
|
61
|
+
implemented ZIP32 reader based on
|
|
62
|
+
[PKWARE's ZIP specification](https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT),
|
|
63
|
+
using the existing
|
|
64
|
+
[pako incremental inflater](https://github.com/nodeca/pako/blob/1.0.6/lib/inflate.js).
|
|
65
|
+
It does not use JSZip's whole-entry `async('string')` allocation path.
|
|
@@ -1,5 +1,77 @@
|
|
|
1
1
|
# Large trace performance follow-up
|
|
2
2
|
|
|
3
|
+
## Long-timeline GPU precision (2026-11-21)
|
|
4
|
+
|
|
5
|
+
The renderer previously uploaded absolute time coordinates as float32. Near
|
|
6
|
+
6,000 seconds in a microsecond recording, endpoints 100 microseconds apart could
|
|
7
|
+
round to the same GPU value, making a call disappear at high zoom. A synthetic
|
|
8
|
+
[late-call recording](../sample/profiles/speedscope/long-late-calls.speedscope.json)
|
|
9
|
+
reproduces that failure, including early and late rectangles in one wide batch.
|
|
10
|
+
|
|
11
|
+
Rectangles are now clipped against the current atlas/source tile using CPU
|
|
12
|
+
doubles, then normalized before upload. The shader transforms unit coordinates
|
|
13
|
+
to the destination. Profile times, range-tree bounds, atlas keys, hit testing,
|
|
14
|
+
frame colors and measured durations retain their existing identities and values.
|
|
15
|
+
Each batch keeps one buffer and one full source-rectangle cache key; a shared
|
|
16
|
+
staging allocation is capped at 1,200,000 bytes for 10,000 rectangles. Switching
|
|
17
|
+
an uncached tile can require another bounded CPU normalization/upload; cached
|
|
18
|
+
tiles and unchanged source rectangles reuse their data. Final profile and GPU
|
|
19
|
+
buffer storage still scale with the recording.
|
|
20
|
+
|
|
21
|
+
Five regressions check real uploaded float32 coordinates and the renderer's
|
|
22
|
+
uniform transform, adjacent late calls, negative/spanning clipping, source
|
|
23
|
+
origin/extent changes, shared staging ownership, buffer lifecycle and limits.
|
|
24
|
+
Browser QA showed both 100-microsecond calls near 6,000 seconds, including after
|
|
25
|
+
overview reset/refocus. The private PLC imported four profiles and rendered
|
|
26
|
+
Time Order, Left Heavy and Sandwich's inverted callers/callees correctly during
|
|
27
|
+
pan/zoom checks, with no browser errors.
|
|
28
|
+
|
|
29
|
+
A warmed Node smoke using a preallocated-copy mock upload and source rectangles
|
|
30
|
+
covering almost all 10,000 rectangles measured about 0.19 ms per repack and
|
|
31
|
+
2.6 ms for 100,000 unchanged-source requests, with one buffer allocation. These are local CPU
|
|
32
|
+
measurements, not browser/GPU frame-rate claims. This addresses the demonstrated
|
|
33
|
+
float32 failure associated with [#467](https://github.com/jlfwong/speedscope/issues/467);
|
|
34
|
+
the original upstream capture remains unavailable for direct comparison. These
|
|
35
|
+
buffer/matrix tests also extend [#125](https://github.com/jlfwong/speedscope/issues/125),
|
|
36
|
+
but do not replace a browser/GPU pixel-regression harness.
|
|
37
|
+
|
|
38
|
+
Final local verification passed all 418 Jest tests in 72 suites with 307
|
|
39
|
+
snapshots on checksum-verified Node 22.23.5 ARM64 using a clean Jest cache.
|
|
40
|
+
The five Node tests, type checking, lint, schema and backlog checks passed
|
|
41
|
+
separately on the system Node runtime. Two full Jest runs on local Node 24.19.0
|
|
42
|
+
ARM64 exited with native access violations without an assertion failure, so a
|
|
43
|
+
green local Node 24 full-suite result is not claimed. GitHub's Windows/Linux
|
|
44
|
+
Node 22/24 matrix validates the branch independently before merge.
|
|
45
|
+
|
|
46
|
+
## Fixed-height Sandwich list geometry (2026-11-21)
|
|
47
|
+
|
|
48
|
+
Sandwich already rendered a viewport window. It now calculates that window and
|
|
49
|
+
initial scroll offset directly from row count/height, removing the size object
|
|
50
|
+
per frame and full-list geometry scans. A single cached selected-frame/index
|
|
51
|
+
replaces the frame-index Map: repeated, adjacent and wrap-around navigation are
|
|
52
|
+
constant time; unrelated new selections scan the existing sorted array once.
|
|
53
|
+
Missing results are cached. Element resizing updates the viewport and filtering
|
|
54
|
+
clamps stale scroll offsets, even without a browser-generated scroll event.
|
|
55
|
+
|
|
56
|
+
Tests cover 100,000-row selection/navigation, billion-row arithmetic without a
|
|
57
|
+
row allocation, visible-row bounds, filter shrink, resize and cleanup. The
|
|
58
|
+
billion-row case tests arithmetic only: browser scroll-height limits still apply.
|
|
59
|
+
The sorted row array, model, sorting and graph/color geometry remain proportional
|
|
60
|
+
to the recording. These changes reduce auxiliary storage and repeated work;
|
|
61
|
+
they do not establish a fully bounded-memory viewer.
|
|
62
|
+
|
|
63
|
+
Combined validation passed 410 Jest tests in 71 suites (307 snapshots), five
|
|
64
|
+
Node tests, type checking, lint, schema/backlog checks and both builds. The
|
|
65
|
+
private PLC gzip still exported four profiles, 17,510 frames and 1,039,656 call
|
|
66
|
+
events with the established SHA-256
|
|
67
|
+
`522de77afbfc4dd94f1f30c3a9551979d12db506305ffef999db9c910ae8eddb`.
|
|
68
|
+
The local Node streaming smoke took about 4.3 seconds to import; this is not a
|
|
69
|
+
browser performance guarantee or a claim of faster importing in this batch.
|
|
70
|
+
In browser QA the first PLC profile's 16,231-frame table rendered 40 rows at the
|
|
71
|
+
scroll end, clamped to zero when filtered to one row, and updated to 49 rendered
|
|
72
|
+
rows after a viewport resize. A generated local function link restored its
|
|
73
|
+
selection after reopening the same PLC recording in another tab.
|
|
74
|
+
|
|
3
75
|
Baseline: `eb93425`, reviewed 2026-11-18. These changes continue the open-issue
|
|
4
76
|
review, prioritizing large profiles and the export request in issue #89.
|
|
5
77
|
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
# OpenTelemetry Profiles JSON
|
|
2
|
+
|
|
3
|
+
The importer accepts the lower-camel-case JSON representation of current
|
|
4
|
+
`opentelemetry.proto.profiles.v1development.ProfilesData` and
|
|
5
|
+
`collector.profiles.v1development.ExportProfilesServiceRequest`. Both have
|
|
6
|
+
`resourceProfiles` and one shared `dictionary`. This is **sampled stack data**,
|
|
7
|
+
not OpenTelemetry distributed traces (`resourceSpans`).
|
|
8
|
+
|
|
9
|
+
The supported schema is pinned to OpenTelemetry Proto commit
|
|
10
|
+
[`d31f7acf9d461fd0601077de516ac1eb7051422f`](https://github.com/open-telemetry/opentelemetry-proto/blob/d31f7acf9d461fd0601077de516ac1eb7051422f/opentelemetry/proto/profiles/v1development/profiles.proto),
|
|
11
|
+
OTLP Profiles **development version 1**. This development schema is unstable.
|
|
12
|
+
Older per-profile dictionaries, `ProfileContainer`, singular `sample`, repeated
|
|
13
|
+
`sampleType`, and future incompatible schemas are unsupported.
|
|
14
|
+
|
|
15
|
+
Development version normally belongs to OTLP transport metadata, not JSON.
|
|
16
|
+
The transport specification defaults absent metadata to version 1. A standalone
|
|
17
|
+
file can optionally advertise `developmentVersion: 1`; other advertised values
|
|
18
|
+
are rejected. This annotation is an importer extension, not a standard schema
|
|
19
|
+
field. Unversioned files must match the documented current dictionary layout;
|
|
20
|
+
the importer cannot identify future incompatible changes that retain that shape.
|
|
21
|
+
|
|
22
|
+
## Dictionary and stacks
|
|
23
|
+
|
|
24
|
+
All seven tables must be present: `stringTable`, `mappingTable`, `locationTable`,
|
|
25
|
+
`functionTable`, `linkTable`, `attributeTable`, and `stackTable`. The official
|
|
26
|
+
schema requires a default-value entry at index zero even in unused tables.
|
|
27
|
+
Default fields inside these entries can be omitted. Link-zero trace/span IDs
|
|
28
|
+
may also be all-zero hexadecimal IDs. References are nonnegative in-range
|
|
29
|
+
int32 indices; zero represents an unset reference.
|
|
30
|
+
|
|
31
|
+
`stackTable[].locationIndices` and inline `locationTable[].lines` are leaf-first;
|
|
32
|
+
the viewer reverses both to retain the full caller/inline chain. Function names
|
|
33
|
+
fall back to system names. File, line and column identify symbolized frames.
|
|
34
|
+
Unsymbolized locations retain their binary filename and exact uint64 hexadecimal
|
|
35
|
+
address. Equal frame values merge independently of dictionary index.
|
|
36
|
+
|
|
37
|
+
Profile/sample `attributeIndices` resolve through `attributeTable`.
|
|
38
|
+
Resource/scope attributes accept direct keys or `keyStrindex`, and values accept
|
|
39
|
+
direct strings, `stringValueStrindex`, exact int64 integers and booleans. Complex
|
|
40
|
+
attributes are ignored for grouping. Duplicate attribute keys are rejected.
|
|
41
|
+
Sample `thread.id`/`thread.name` override profile attributes and create separate
|
|
42
|
+
thread profiles. Integer thread IDs remain exact decimal strings. Input profile
|
|
43
|
+
objects, resources and scopes remain separate even when names or IDs match;
|
|
44
|
+
labels include the service (or executable), scope, metric, profile ordinal and
|
|
45
|
+
thread. Generic attributes, scope versions, profile IDs and span links are not
|
|
46
|
+
preserved in viewer/export metadata. Links never become stack frames.
|
|
47
|
+
|
|
48
|
+
## Values, units and time
|
|
49
|
+
|
|
50
|
+
Each sample needs `values`, `timestampsUnixNano`, or both. Both populated arrays
|
|
51
|
+
must have equal length. Values sum directly; timestamp-only observations each
|
|
52
|
+
contribute **one in the declared sample unit**, exactly as specified by OTLP.
|
|
53
|
+
`period` and `periodType` are informational and never multiply weights.
|
|
54
|
+
|
|
55
|
+
Supported units are `nanoseconds`/`ns`, `microseconds`/`us`/`µs`/`μs`,
|
|
56
|
+
`milliseconds`/`ms`, `seconds`/`s`, `bytes`/`By`, and raw counts
|
|
57
|
+
(`count`, `1`, `{count}`, or empty unit). Unknown units are rejected. Values are
|
|
58
|
+
not rescaled when selecting the formatter. Negative differential weights and
|
|
59
|
+
weights/totals exceeding JavaScript's exact integer range are rejected.
|
|
60
|
+
|
|
61
|
+
Profiles are imported as **aggregate**, with measured stack weights. Epoch
|
|
62
|
+
timestamps do not become a duration axis, and gaps do not generate idle stacks.
|
|
63
|
+
Timestamps and addresses use exact uint64 parsing; decimal strings are required
|
|
64
|
+
for values beyond JavaScript's safe integer range. Timestamp validation uses
|
|
65
|
+
the exact epoch. A nonzero profile duration requires samples within the declared
|
|
66
|
+
half-open time range; zero-duration instant profiles skip this range check.
|
|
67
|
+
Overflowing time ranges are rejected. Timestamps and period metadata are not
|
|
68
|
+
retained in exports.
|
|
69
|
+
|
|
70
|
+
## Limits and remaining support
|
|
71
|
+
|
|
72
|
+
The JSON document is currently parsed in memory; this is not a streaming JSON or
|
|
73
|
+
protobuf importer. Import processing avoids an expanded observation/record array
|
|
74
|
+
and does not retain flattened copies of every dictionary stack. Memory consists
|
|
75
|
+
of the parsed JSON, output profiles, per-thread builder state and one stack.
|
|
76
|
+
Main-thread processing yields every 4,096 counted operations and checks
|
|
77
|
+
cancellation. The dedicated import worker explicitly disables these timer
|
|
78
|
+
yields, because cancelling the worker terminates its execution externally.
|
|
79
|
+
The worker keeps the same limits and weight checks. Main-thread fallback keeps
|
|
80
|
+
cooperative yielding, including when no cancellation signal was supplied; the
|
|
81
|
+
optimization never infers worker execution from the presence of a signal.
|
|
82
|
+
|
|
83
|
+
Limits are one million dictionary entries total; one million elements per input
|
|
84
|
+
array; 100,000 expanded frames per stack; 4,096 attributes per set; 4,096
|
|
85
|
+
output profiles; 4,096 characters per grouping attribute value; and ten million
|
|
86
|
+
counted hierarchy/sample/observation/location/inline operations. Limits fail
|
|
87
|
+
explicitly rather than silently dropping samples. The parsed JSON and output
|
|
88
|
+
itself still consume memory proportional to their size.
|
|
89
|
+
|
|
90
|
+
Binary OTLP protobuf, old development layouts, original payload decoding,
|
|
91
|
+
automatic network collection, and chronological OTLP visualization remain
|
|
92
|
+
unsupported. Binary protobuf has no unique identifying magic; the importer does
|
|
93
|
+
not guess it ahead of existing pprof detection. The synthetic fixture and its
|
|
94
|
+
provenance are in `sample/profiles/otel-profiles`.
|
|
95
|
+
|
|
96
|
+
## Worker import performance check
|
|
97
|
+
|
|
98
|
+
On Windows on 2026-11-21, two fresh Node processes per case imported generated
|
|
99
|
+
current-schema JSON based on `synthetic.json`: its samples were replaced with
|
|
100
|
+
100,000 or 1,000,000 distinct `{stackIndex: 1, values: [String(i % 3 + 1)]}`
|
|
101
|
+
objects. This exercises ordinary JSON detection and parsing, not a pre-parsed
|
|
102
|
+
importer call. The benchmark invoked `processTraceInWorker` directly, including
|
|
103
|
+
snapshot construction; it excludes browser worker startup and message transfer.
|
|
104
|
+
The before case restored timer yielding in the worker options only.
|
|
105
|
+
|
|
106
|
+
| Samples | JSON size | Main thread | Worker before | Worker without timer yields |
|
|
107
|
+
| --------- | --------- | ------------- | ------------- | --------------------------- |
|
|
108
|
+
| 100,000 | 3.05 MiB | 1.82–1.91 s | 1.94 s | 0.159–0.170 s |
|
|
109
|
+
| 1,000,000 | 30.52 MiB | 19.00–19.46 s | 19.51–19.75 s | 1.396–1.399 s |
|
|
110
|
+
|
|
111
|
+
The worker removes 122 and 1,220 timer waits respectively. Totals remained
|
|
112
|
+
199,999 and 1,999,999 nanoseconds. Main-thread yielding remains enabled.
|
|
113
|
+
This is a timing optimization, not a new memory reduction: process high-water
|
|
114
|
+
RSS for the million-sample runs was 359–382 MiB before and 392–402 MiB after,
|
|
115
|
+
including fixture generation and all Node runtime allocations. Post-import
|
|
116
|
+
retained heap was comparable. There is no additional observation array or stack
|
|
117
|
+
cache; whole-document JSON parsing remains the dominant memory limitation.
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# Linux perf imports and weight semantics
|
|
2
|
+
|
|
3
|
+
Import text from `perf script`. Include `comm`, thread/process identity, `time`,
|
|
4
|
+
`event`, `period`, `ip`, `sym` and `dso` fields when available. Regular callchains
|
|
5
|
+
and leaf-only `--hide-call-graph` output are supported. Keeping `period` is
|
|
6
|
+
important: timestamps alone do not measure a sampled function's CPU consumption.
|
|
7
|
+
|
|
8
|
+
## Periods determine the weights
|
|
9
|
+
|
|
10
|
+
| Recorded event | Period present | Period absent |
|
|
11
|
+
| --- | --- | --- |
|
|
12
|
+
| `cpu-clock`, `task-clock` | Aggregate sampled CPU time, in nanoseconds | Legacy inferred timestamp timeline, explicitly labeled |
|
|
13
|
+
| `offcpu-time` | Aggregate sampled off-CPU time, in nanoseconds | Sample count, explicitly labeled |
|
|
14
|
+
| `cycles`, including hybrid CPU PMUs | Aggregate raw cycle periods | Legacy inferred timestamp timeline, explicitly labeled |
|
|
15
|
+
| Other hardware/software counters | Aggregate raw event periods | Sample count, explicitly labeled |
|
|
16
|
+
|
|
17
|
+
Event modifiers such as `:u` retain these rules. Hybrid `cpu_core/cycles/u` and
|
|
18
|
+
`cpu_atom/cycles/u` samples are combined under the normalized `cycles:u` event.
|
|
19
|
+
Different events, processes and threads remain separate. If some records have
|
|
20
|
+
periods while others omit them, their profiles remain separate too, so inferred
|
|
21
|
+
seconds, nanosecond periods and counts are never added together.
|
|
22
|
+
|
|
23
|
+
Period-weighted profiles use Left Heavy, Sandwich or Call Tree. Time Order is
|
|
24
|
+
disabled because accumulating sampled weights does not establish exact wall-time
|
|
25
|
+
intervals. Missing-period clock/cycle profiles retain the older timestamp
|
|
26
|
+
interpolation and are labeled **inferred timeline, period missing**. That view
|
|
27
|
+
assigns gaps between timestamps to adjacent stacks; do not interpret it as
|
|
28
|
+
measured CPU time or proof that a thread was running throughout those gaps.
|
|
29
|
+
|
|
30
|
+
Periods must be finite, non-negative integers within JavaScript's exact range
|
|
31
|
+
(`0` through `9007199254740991`). Larger individual periods fail with an
|
|
32
|
+
actionable error rather than being silently rounded. Totals still use the
|
|
33
|
+
project's ordinary floating-point weight model. Zero periods contribute zero
|
|
34
|
+
weight; they do not trigger timestamp inference or become a one-unit sample.
|
|
35
|
+
|
|
36
|
+
## Why gaps are not CPU time
|
|
37
|
+
|
|
38
|
+
The public reproduction in [upstream #393](https://github.com/jlfwong/speedscope/issues/393#issuecomment-1200547074)
|
|
39
|
+
contains 13 samples for `DiscoveryClient` thread `2614953`, each with a
|
|
40
|
+
`cpu-clock` period of `20408163` nanoseconds. Its sampled CPU weight is therefore
|
|
41
|
+
`265306119` ns, about **0.265 seconds**, while its timestamps span about
|
|
42
|
+
**30.231 seconds**. The importer now uses the periods instead of attributing
|
|
43
|
+
those gaps to functions. A single clock sample also retains its real period
|
|
44
|
+
instead of becoming an arbitrary one-unit sample.
|
|
45
|
+
|
|
46
|
+
The checked-in [regression fixture](../sample/profiles/linux-perf/idle-clock-period.linux-perf.txt)
|
|
47
|
+
uses those timestamp/period facts with authored synthetic stacks. The source
|
|
48
|
+
archive was [profile.out.gz](https://github.com/jlfwong/speedscope/files/9230041/profile.out.gz),
|
|
49
|
+
22,149 bytes, SHA-256
|
|
50
|
+
`40b9123046dc918018f40030426851d15622077843a8cd2923371750a6fa5efa`.
|
|
51
|
+
|
|
52
|
+
This fixes sampled aggregate attribution. Exact idle-gap placement and aligned
|
|
53
|
+
multithread timelines remain separate work: clock periods do not say exactly
|
|
54
|
+
when each fragment of CPU execution occurred within a timestamp gap. An
|
|
55
|
+
`offcpu-time` profile is kept separate and never added to CPU execution weights.
|
|
56
|
+
Scheduler-switch events and their associations need their own tested timeline
|
|
57
|
+
mapping; the importer does not invent those intervals.
|
|
58
|
+
|
|
59
|
+
## Format references
|
|
60
|
+
|
|
61
|
+
Linux's [software-clock sampling implementation](https://github.com/torvalds/linux/blob/master/kernel/events/core.c)
|
|
62
|
+
initializes clock periods in nanoseconds and arms timers with
|
|
63
|
+
`ns_to_ktime(period)`. The kernel's [perf record documentation](https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf-record.txt)
|
|
64
|
+
specifies that `--off-cpu` produces `offcpu-time` samples whose periods are sleep
|
|
65
|
+
time in nanoseconds. Other event periods remain counts in that event's own
|
|
66
|
+
units; cycle counts are not converted to seconds.
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# pprof and signed heap deltas
|
|
2
|
+
|
|
3
|
+
Import binary protobuf pprof, including its usual gzip wrapper. The selected
|
|
4
|
+
metric is `default_sample_type`, or the last sample type when no usable default
|
|
5
|
+
is present. Positive-only recordings retain the ordinary single aggregate
|
|
6
|
+
profile. Time Order is unavailable: protobuf samples do not establish call order.
|
|
7
|
+
|
|
8
|
+
Go's `/debug/pprof/heap?seconds=N` can return a delta recording containing
|
|
9
|
+
negative values. Those values now import as two explicitly named profiles:
|
|
10
|
+
|
|
11
|
+
- **Negative contributions (magnitude; subtract)** contains the absolute values
|
|
12
|
+
of negative samples.
|
|
13
|
+
- **Positive contributions** contains positive samples.
|
|
14
|
+
|
|
15
|
+
These are signed contributions, not historical before/after heap snapshots.
|
|
16
|
+
Choose **Profile actions → Compare profiles…**, with negative contributions as
|
|
17
|
+
Baseline and positive contributions as Candidate, in **Absolute weights** mode.
|
|
18
|
+
Candidate minus baseline gives the net signed change for each call path. The
|
|
19
|
+
same-path contributions remain separate until comparison; cancellation to zero
|
|
20
|
+
does not discard either side. The graph's union widths represent gross
|
|
21
|
+
contribution magnitudes, not elapsed time or live-heap size. Normalized mode
|
|
22
|
+
instead compares relative composition and should not be read as net byte change.
|
|
23
|
+
|
|
24
|
+
For example, samples of -9, +12 and -2 bytes on one stack become 11 negative and
|
|
25
|
+
12 positive bytes, yielding +1 net byte. All-negative recordings have an empty
|
|
26
|
+
positive side and remain comparable. Zero-valued samples have no width. Export
|
|
27
|
+
preserves both magnitude profiles in ordinary speedscope JSON; their names
|
|
28
|
+
describe the projection. Export does not recreate the original signed protobuf.
|
|
29
|
+
|
|
30
|
+
Selected sample values and the total on each side must fit exact JavaScript
|
|
31
|
+
integers (`Number.MAX_SAFE_INTEGER`). Larger values fail explicitly rather than
|
|
32
|
+
rounding or dropping negative values. This is not arbitrary-precision int64
|
|
33
|
+
support. Existing pprof limitations remain: only the selected metric is imported,
|
|
34
|
+
only the outermost inline location frame is retained, labels and mapping/address
|
|
35
|
+
metadata are not fully represented, and parsed input plus output trees scale
|
|
36
|
+
with capture size. Generic gzip decompression is not bounded by the text-stream
|
|
37
|
+
limits. This change does not make pprof streaming or out of core.
|
|
38
|
+
|
|
39
|
+
A local Go 1.24.1 `net/http/pprof` heap-delta check allocated then released 16 MiB
|
|
40
|
+
between collections. It imported 16,777,600 negative and 78,112 positive bytes,
|
|
41
|
+
giving -16,699,488 bytes net; browser comparison displayed -15.93 MiB. Allocation
|
|
42
|
+
overhead varies between runs. The ignored capture is an authored regression
|
|
43
|
+
observation, not the unavailable original upstream recording. Deterministic
|
|
44
|
+
checked-in tests cover both signs, same-path cancellation, all-negative values,
|
|
45
|
+
unit preservation, exact-range failures and export round trips.
|
|
46
|
+
|
|
47
|
+
- [Go HTTP pprof parameters](https://pkg.go.dev/net/http/pprof#hdr-Parameters)
|
|
48
|
+
define `seconds=N` delta recording behavior.
|
|
49
|
+
- [pprof protobuf schema](https://github.com/google/pprof/blob/main/proto/profile.proto)
|
|
50
|
+
defines sample type selection and signed int64 sample values.
|