Real-time spectral processing: freeze, blur, morph, spectral filtering and cross-synthesis, on an audio path that has no FFT #140

New issue

Open

opened 2026-10-06 00:33:31 +00:00 by bcox · 0 comments

bcox commented

2026-10-06 00:33:31 +00:00

Owner

bcox asked on 2026-10-05 for a ticket for each family in his gaps.md that has none. Engine claims are from reading the tree at cd68bd3e on main; literature named below is from recollection and was not re-read. Nothing here is built, measured or listened to.

What the tree has

No FFT on the audio path. dsp/ has none; the string appears there only inside test names.
The engine's own spectral representation is additive: dsp/spectrum.h's Wave is a list of harmonic amplitudes and phases, and a SpectralTable is a sequence of them crossfaded harmonic by harmonic, which the FM pair plays. resynth/ produces those tables offline, in double, from a recording; it is analysis as a build step, not a live engine (#16, closed).
The partial bank (PartialLayer) plays a trajectory and takes no input; #92 is its noisy residual.
The audio arrives. dsp/mod_source.h's kSharedExternalInput and kSharedSidechain reach every voice, each engine's primary input selector can read them (WaveguideLayer::Drive::kInput, the VA oscillator's tap), and the console's compressor has a detector (fabric/inserts.h). So an incoming signal can excite, modulate and be filtered. Nothing analyzes it in the frequency domain.
#105 lists a vocoder as a candidate layer engine: a 16 to 32 band time-domain one, filter banks and followers, no FFT. #118 is the three control sources from audio (follower, trigger, pitch).

What is missing

Spectral freeze (hold one frame's magnitudes and resynthesize with running phase); spectral blur and smear; a morph between two live analyses; spectral filtering and gating by a drawn or derived mask; cross-synthesis, the spectral envelope of one signal imposed on another, which is what a phase-vocoder vocoder does and a band vocoder approximates. Time-stretch and formant-preserving pitch shift are the granular ticket's (#141), which can do them either way.

Two shapes, and why the second is the engine's own

An STFT engine. A real FFT of 512 to 2048 points per hop per channel, processing in the bins, an inverse and overlap-add. On fabric a pipelined FFT is a standard block; the price is M20K for frames and twiddles and DSPs for butterflies, and the latency is one window, fixed and announced, which is rule zero kept rather than broken. It is channel-scope by nature, one input per channel, so a channel-scope layer like the long delay and the reverb, with its frames in a dsp::StorageRegion block. The fixed-point question is word growth: a radix-2 stage grows a bit, which dsp/fixed.h makes explicit, and the rounding mode per stage decides the noise floor.
A live analyzer feeding the tables the engine already plays. A per-channel harmonic analysis producing one SpectralTable entry per hop, handed to an FM pair or partial bank as a table that moves: a resynth running at the output rate, pitch-tracked (#118's tracker). Freeze is the analyzer stopping; morph is the crossfade the table already does; spectral filtering is a mask on the harmonic amplitudes, a matrix destination per band. It is what dsp/spectrum.h's own comment anticipates ("a Synclavier-style resynthesis frame drops straight in as one entry"), it changes no renderer, and it is harmonic-only. The analysis itself is where the FFT sits, or a bank of heterodyne filters at the tracked harmonics, which is what Beauchamp's SNDAN did and which costs one complex multiply-add per harmonic per sample, no transform.

The second plays to what the engine has and is the recommended first step; the first is what every other instrument means by "spectral" and should be priced in fabric before either is designed.

Open

Frame size against latency, and whether a channel's latency is reported to the console so the strips can align.
Whether a live table's phases are kept (resynthesis) or free-run (freeze), which is the difference between a rendition and a drone.
Where the inharmonic residual goes: #92's bandwidth-enhanced partials, or the unvoiced layer driven by the analyzer's noise estimate.

#105 (band vocoder), #118, #92, #16 and #20 (closed, the offline analysis and the spectral timeline), #141.

bcox asked on 2026-10-05 for a ticket for each family in his `gaps.md` that has none. Engine claims are from reading the tree at `cd68bd3e` on `main`; literature named below is from recollection and was not re-read. **Nothing here is built, measured or listened to.** ## What the tree has - No FFT on the audio path. `dsp/` has none; the string appears there only inside test names. - The engine's own spectral representation is additive: `dsp/spectrum.h`'s `Wave` is a list of harmonic amplitudes and phases, and a `SpectralTable` is a sequence of them crossfaded harmonic by harmonic, which the FM pair plays. `resynth/` produces those tables offline, in `double`, from a recording; it is analysis as a build step, not a live engine (#16, closed). - The partial bank (`PartialLayer`) plays a trajectory and takes no input; #92 is its noisy residual. - The audio arrives. `dsp/mod_source.h`'s `kSharedExternalInput` and `kSharedSidechain` reach every voice, each engine's primary input selector can read them (`WaveguideLayer::Drive::kInput`, the VA oscillator's tap), and the console's compressor has a detector (`fabric/inserts.h`). So an incoming signal can excite, modulate and be filtered. Nothing analyzes it in the frequency domain. - #105 lists a vocoder as a candidate layer engine: a 16 to 32 band time-domain one, filter banks and followers, no FFT. #118 is the three control sources from audio (follower, trigger, pitch). ## What is missing Spectral freeze (hold one frame's magnitudes and resynthesize with running phase); spectral blur and smear; a morph between two live analyses; spectral filtering and gating by a drawn or derived mask; cross-synthesis, the spectral envelope of one signal imposed on another, which is what a phase-vocoder vocoder does and a band vocoder approximates. Time-stretch and formant-preserving pitch shift are the granular ticket's (#141), which can do them either way. ## Two shapes, and why the second is the engine's own 1. **An STFT engine.** A real FFT of 512 to 2048 points per hop per channel, processing in the bins, an inverse and overlap-add. On fabric a pipelined FFT is a standard block; the price is M20K for frames and twiddles and DSPs for butterflies, and the latency is one window, fixed and announced, which is rule zero kept rather than broken. It is channel-scope by nature, one input per channel, so a channel-scope layer like the long delay and the reverb, with its frames in a `dsp::StorageRegion` block. The fixed-point question is word growth: a radix-2 stage grows a bit, which `dsp/fixed.h` makes explicit, and the rounding mode per stage decides the noise floor. 2. **A live analyzer feeding the tables the engine already plays.** A per-channel harmonic analysis producing one `SpectralTable` entry per hop, handed to an FM pair or partial bank as a table that moves: a `resynth` running at the output rate, pitch-tracked (#118's tracker). Freeze is the analyzer stopping; morph is the crossfade the table already does; spectral filtering is a mask on the harmonic amplitudes, a matrix destination per band. It is what `dsp/spectrum.h`'s own comment anticipates ("a Synclavier-style resynthesis frame drops straight in as one entry"), it changes no renderer, and it is harmonic-only. The analysis itself is where the FFT sits, or a bank of heterodyne filters at the tracked harmonics, which is what Beauchamp's SNDAN did and which costs one complex multiply-add per harmonic per sample, no transform. The second plays to what the engine has and is the recommended first step; the first is what every other instrument means by "spectral" and should be priced in fabric before either is designed. ## Open - Frame size against latency, and whether a channel's latency is reported to the console so the strips can align. - Whether a live table's phases are kept (resynthesis) or free-run (freeze), which is the difference between a rendition and a drone. - Where the inharmonic residual goes: #92's bandwidth-enhanced partials, or the unvoiced layer driven by the analyzer's noise estimate. ## Related #105 (band vocoder), #118, #92, #16 and #20 (closed, the offline analysis and the spectral timeline), #141.