bcox/tymbal

Fork 0

//dsp:modal_loop_test is the ASan lane's critical path at 74% of its budget #61

New issue

Closed

opened 2026-09-05 12:31:37 +00:00 by bcox · 3 comments

bcox commented

2026-09-05 12:31:37 +00:00

Owner

After modal-cymbals sharded //patches:patches_test, the slowest target in
the ASan lane is //dsp:modal_loop_test at 665 s against a 900 s budget
(size = "large", dsp/BUILD.bazel:240) -- 74%, and unsharded.

For contrast, //patches:patches_test now runs four shards at 262 / 344 / 364 /
399 s, so its worst shard sits at 44% of its own budget. The whole ASan lane is
734 s wall.

Nothing on modal-cymbals touched dsp/modal_loop_test.cc; it simply became
the binding target once the other one was sharded. Filed so that the next
person to add a case to it knows what the headroom is, rather than finding out
from a timeout.

shard_count is the same one-line remedy that worked for the patches target,
and it needs no code movement. Worth knowing from that experience: sharding is
not free -- four ASan shards of patches_test summed to 1.53x the single-
process time, because ASan renderers contend for memory bandwidth. What it buys
is N budgets, not less work.

Found while running the sanitizer lanes for modal-cymbals.

After `modal-cymbals` sharded `//patches:patches_test`, the slowest target in the ASan lane is `//dsp:modal_loop_test` at **665 s against a 900 s budget** (`size = "large"`, `dsp/BUILD.bazel:240`) -- 74%, and unsharded. For contrast, `//patches:patches_test` now runs four shards at 262 / 344 / 364 / 399 s, so its worst shard sits at 44% of its own budget. The whole ASan lane is 734 s wall. Nothing on `modal-cymbals` touched `dsp/modal_loop_test.cc`; it simply became the binding target once the other one was sharded. Filed so that the next person to add a case to it knows what the headroom is, rather than finding out from a timeout. `shard_count` is the same one-line remedy that worked for the patches target, and it needs no code movement. Worth knowing from that experience: sharding is not free -- four ASan shards of `patches_test` summed to 1.53x the single- process time, because ASan renderers contend for memory bandwidth. What it buys is N budgets, not less work. Found while running the sanitizer lanes for `modal-cymbals`.

bcox commented

2026-09-09 22:27:34 +00:00

Author

Owner

The asan lane has moved past "critical path at 74% of its budget" to actually timing out. Measured on ladder-gate at 2026-09-09, running bazel test --config=asan //... with default timeouts:

target	declared	under the lane	alone, `--test_timeout=3600`
`//dsp:modal_loop_test`	900 s	TIMEOUT	549.3 s (61%)
`//editor:patch_view_id_conflict_test`	300 s	TIMEOUT	135.0 s (45%)
`//plugin:patch_format_test`	900 s	TIMEOUT	664.8 s (74%)

All three fit their budgets comfortably alone; none fits under the lane's own 16-way parallelism. The whole lane passes 98/98 with --test_timeout=3600.

Two things worth recording alongside this issue's current title:

.bazelrc still says asan and ubsan fit inside the default timeouts. That has stopped being true.
plugin/BUILD.bazel records patch_format_test at 220.0 s under asan on 2026-09-05, against 664.8 s now — 3x in four days. That comment's own text asks for a re-measure rather than trust, which is what this is.

Not caused by ladder-gate, and left unfixed there deliberately: resizing a target or widening a budget is a policy call rather than a branch's business.

Worth noting for whoever takes it: ladder-gate adds ~58 s to //dsp:va_filter_test, already size = "large" at ~140 s unsanitized, which is plausibly the next target to tip. dsp/BUILD.bazel documents that cost and names the lever — the onset bisection's step count — rather than shrinking the test.

Three sibling worktrees had live Bazel servers on the machine throughout, which is part of the contention and is also the normal working condition here.

The asan lane has moved past "critical path at 74% of its budget" to actually timing out. Measured on `ladder-gate` at 2026-09-09, running `bazel test --config=asan //...` with default timeouts: | target | declared | under the lane | alone, `--test_timeout=3600` | |---|---|---|---| | `//dsp:modal_loop_test` | 900 s | TIMEOUT | 549.3 s (61%) | | `//editor:patch_view_id_conflict_test` | 300 s | TIMEOUT | 135.0 s (45%) | | `//plugin:patch_format_test` | 900 s | TIMEOUT | 664.8 s (74%) | All three fit their budgets comfortably alone; none fits under the lane's own 16-way parallelism. The whole lane passes 98/98 with `--test_timeout=3600`. Two things worth recording alongside this issue's current title: - `.bazelrc` still says asan and ubsan fit inside the default timeouts. That has stopped being true. - `plugin/BUILD.bazel` records `patch_format_test` at 220.0 s under asan on 2026-09-05, against 664.8 s now — 3x in four days. That comment's own text asks for a re-measure rather than trust, which is what this is. Not caused by `ladder-gate`, and left unfixed there deliberately: resizing a target or widening a budget is a policy call rather than a branch's business. Worth noting for whoever takes it: `ladder-gate` adds ~58 s to `//dsp:va_filter_test`, already `size = "large"` at ~140 s unsanitized, which is plausibly the next target to tip. `dsp/BUILD.bazel` documents that cost and names the lever — the onset bisection's step count — rather than shrinking the test. Three sibling worktrees had live Bazel servers on the machine throughout, which is part of the contention and is also the normal working condition here.

bcox referenced this issue from a commit

2026-09-10 05:18:09 +00:00

The truncation sweeps stop building a voice pool per byte

bcox referenced this issue

2026-09-10 18:52:17 +00:00

Ladder gate execution ledger: 10 rulings, a bug the spec called a characteristic, and five classes of a self-contradicting comment #84

bcox commented

2026-09-10 20:23:22 +00:00

Author

Owner

Closed by removing the cause rather than by sharding. //dsp:modal_loop_test now reads 8s under asan, against the 665s this issue measured.

The premise was right and the remedy would not have helped much: as this issue's own text says, shard_count buys N budgets, not less work. The work was the problem, and it was not the tests.

.bazelrc set no -O at all for the default build or for either sanitizer lane -- only --config=opt carried one. This engine is header-only fixed point, so at -O0 every Sample operation is an un-inlined template call. That is worth about 30x here rather than the usual small constant. build --cxxopt=-O1 (commit c43da4f):

	before	after
`//dsp:modal_loop_test` (asan)	665 s	8 s
`//dsp:modal_loop_test` (fastbuild)	158 s	4.9 s
its worst single test	89 s	3.1 s
worst asan target in the tree	665 s	43 s
slowest single test anywhere	89 s	6.4 s
whole suite, fastbuild, uncached	--	28 s

All five lanes green at c43da4f: fastbuild, --config=opt, asan, ubsan all 101/101, tsan 1/1. Goldens unmoved -- and the repo already had the check that guarantees that, since goldens must match at fastbuild and --config=opt, so an optimization level that changed a result would be a failing test.

-O1 rather than -O2: it is what restores inlining, it is what ASan's own documentation recommends alongside the -g and -fno-omit-frame-pointer the lanes already pass, and it keeps a debugger's view of locals mostly intact. The cost is written into .bazelrc rather than glossed -- an optimizer can delete a computation before UBSan instruments it, so -O1 can hide UB in code whose result is unused. --cxxopt=-O0 on the command line overrides it for a specific hunt.

Four targets drop large -> medium. Every justification for large was arithmetic against an -O0 reading -- 220s of patch_format_test, 256s of modal_layer_test, 665s of this one, 1369.3s summed across four patches_test shards -- and each now reads 8 to 15s under asan. The durable halves of those comments are kept, including this issue's own contention finding: sharding buys budgets, not less work.

patches_test keeps its four shards. They are no longer needed for the budget -- one process would fit medium comfortably -- and churning them buys nothing.

One thing deliberately not done: bazel now warns that most targets could be size = "small". Those are fastbuild readings, and the binding lane is asan, where the worst target is 43s against small's 60s. That is the same 73%-headroom trap patch_format_test's own comment talked itself out of once -- correct reasoning, stale input.

Closed by removing the cause rather than by sharding. `//dsp:modal_loop_test` now reads **8s under asan**, against the 665s this issue measured. The premise was right and the remedy would not have helped much: as this issue's own text says, `shard_count` buys N budgets, not less work. The work was the problem, and it was not the tests. `.bazelrc` set **no `-O` at all** for the default build or for either sanitizer lane -- only `--config=opt` carried one. This engine is header-only fixed point, so at `-O0` every `Sample` operation is an un-inlined template call. That is worth about 30x here rather than the usual small constant. `build --cxxopt=-O1` (commit c43da4f): | | before | after | |---|---|---| | `//dsp:modal_loop_test` (asan) | 665 s | 8 s | | `//dsp:modal_loop_test` (fastbuild) | 158 s | 4.9 s | | its worst single test | 89 s | 3.1 s | | worst asan target in the tree | 665 s | 43 s | | slowest single test anywhere | 89 s | 6.4 s | | **whole suite, fastbuild, uncached** | -- | **28 s** | All five lanes green at `c43da4f`: fastbuild, `--config=opt`, asan, ubsan all 101/101, tsan 1/1. Goldens unmoved -- and the repo already had the check that guarantees that, since goldens must match at fastbuild and `--config=opt`, so an optimization level that changed a result would be a failing test. `-O1` rather than `-O2`: it is what restores inlining, it is what ASan's own documentation recommends alongside the `-g` and `-fno-omit-frame-pointer` the lanes already pass, and it keeps a debugger's view of locals mostly intact. The cost is written into `.bazelrc` rather than glossed -- an optimizer can delete a computation before UBSan instruments it, so `-O1` can hide UB in code whose result is unused. `--cxxopt=-O0` on the command line overrides it for a specific hunt. Four targets drop `large` -> `medium`. Every justification for `large` was arithmetic against an `-O0` reading -- 220s of `patch_format_test`, 256s of `modal_layer_test`, 665s of this one, 1369.3s summed across four `patches_test` shards -- and each now reads 8 to 15s under asan. The durable halves of those comments are kept, including this issue's own contention finding: sharding buys budgets, not less work. `patches_test` keeps its four shards. They are no longer needed for the budget -- one process would fit `medium` comfortably -- and churning them buys nothing. One thing deliberately not done: bazel now warns that most targets could be `size = "small"`. Those are fastbuild readings, and the binding lane is asan, where the worst target is 43s against small's 60s. That is the same 73%-headroom trap `patch_format_test`'s own comment talked itself out of once -- correct reasoning, stale input.

bcox closed this issue

2026-09-10 20:23:22 +00:00

bcox referenced this issue from a commit

2026-09-10 21:26:26 +00:00

Build at -O1 by default, and stop sizing tests around -O0

bcox commented

2026-09-10 21:29:15 +00:00

Author

Owner

Correcting my own comment above, which went stale two commits after I wrote it.

It says the default is -O1 and cites c43da4f as the final state. The default is now -O3 (07a725d) -- bcox asked for the shipping optimization everywhere rather than a step toward it. Deliberately the optimization only and not -c opt, since that half defines NDEBUG and would compile out eight runtime assert()s, one of which is the net dsp/va_filter_test.cc's MorphAtMaximumIsTheHighpassOfThatOrder relies on in the default lane.

Re-measured at -O3, so the figures in that comment read low:

	before	-O1	-O3
`//dsp:modal_loop_test` (asan)	665 s	8 s	11 s
worst asan target in the tree	665 s	43 s	23 s

The worst asan target is //resynth:pipeline_test and //editor:patch_view_id_conflict_test, both 23 s. modal_loop_test -- this issue's subject -- is 11 s.

The conclusion is unchanged and so is the closure: sharding was never the fix, because the work was the problem and it was not the tests. Everything else in the comment above still holds, including the sizing decision and the reason for not chasing bazel's size = "small" warnings.

All five lanes green at 07a725d: fastbuild, --config=opt, asan, ubsan 101/101, tsan 1/1.

Correcting my own comment above, which went stale two commits after I wrote it. It says the default is `-O1` and cites `c43da4f` as the final state. The default is now **`-O3`** (`07a725d`) -- bcox asked for the shipping optimization everywhere rather than a step toward it. Deliberately the optimization only and not `-c opt`, since that half defines `NDEBUG` and would compile out eight runtime `assert()`s, one of which is the net `dsp/va_filter_test.cc`'s `MorphAtMaximumIsTheHighpassOfThatOrder` relies on in the default lane. Re-measured at `-O3`, so the figures in that comment read low: | | before | -O1 | -O3 | |---|---|---|---| | `//dsp:modal_loop_test` (asan) | 665 s | 8 s | 11 s | | worst asan target in the tree | 665 s | 43 s | 23 s | The worst asan target is `//resynth:pipeline_test` and `//editor:patch_view_id_conflict_test`, both 23 s. `modal_loop_test` -- this issue's subject -- is 11 s. The conclusion is unchanged and so is the closure: sharding was never the fix, because the work was the problem and it was not the tests. Everything else in the comment above still holds, including the sizing decision and the reason for not chasing bazel's `size = "small"` warnings. All five lanes green at `07a725d`: fastbuild, `--config=opt`, asan, ubsan 101/101, tsan 1/1.

bcox referenced this issue

2026-09-10 21:30:55 +00:00

Set up CI, with an opt lane and a sanitizer lane #1