Spectral effects

Transform sound in the frequency domain, including effects that reshape or retain its spectral character.

The live STFT track-effect family: freeze, blur, spectralGate, tilt, spectralMorph. Design spec: Spectral effects. Demo song: examples/songs/hoarfrost.volta.

A sixth mode shares this engine but is documented with its compile-time twin: spectralTune amount (snap each spectral peak onto the module tuning's absolute frequency lattice) — see Match timbre to tuning §4. Everything in §2 below about the STFT layout, the identity-at-0 rule, and the fixed bus latency applies to it unchanged.

1. Surface#

All five are track-level effects (one instance on the mixer bus):

hold = macro "hold" 0.45 (lfoSine @ 1/2c |> scale [0, 0.35])
padTrack = play pad chords |> freeze hold           -- catch & sustain a frame
fog      = play keys mel   |> blur 0.7              -- spectral smear
bones    = play vox line   |> spectralGate 0.35     -- keep bins ≥ 35 % of peak
dark     = play lead riff  |> tilt (0 - 0.6)        -- ±6 dB/oct about 1 kHz
ghost    = play pluck arpPat |> spectralMorph "speech" wisp  -- cross-synthesis

All five are also legal mixer inserts — master or submix (mixer { master = { inserts = [freeze 0.8] } }): an insert lowers through the same track-effect path as a |> pipe, so spectralMorph "speech" 0.5 works there too.

Naming note: this is spectral freeze, unrelated to track-freeze — bouncing a track down to a sample (Resample your own sounds).

2. Semantics#

One streaming STFT per channel — periodic-Hann analysis, FFT 1024, hop 256 (75 % overlap), Hann synthesis, overlap-add ÷ 1.5, the smbPitchShift FIFO layout. Output is always the resynthesis path: at knob = 0 every effect is the identity transform (up to the fixed 1024-sample group delay, ~21 ms at 48 kHz), and wet/dry blending happens in the spectral domain (complex crossfade) so nothing combs against an undelayed dry branch.

• freeze — when amount rises above 0.01, the current frame is captured: per-bin magnitude, phase, and phase-advance (expected hop advance + measured deviation = each partial's true frequency). The frozen frame resynthesises forever by advancing phases; output = complex lerp live→frozen by amount. Dropping the knob back to ≤ 0.01 releases (and re-arms the capture).
• blur — per-bin magnitude one-pole smoothing across frames (coefficient 0.55 + 0.44a) + a seeded random-walk phase offset per bin (step π(0.02 + 0.5a)); output = complex lerp by a. At a = 0 the smoother tracks live magnitudes so a later blur fades in from now.

• spectralGate — per-bin Wiener gain mag² / (mag² + cut²) against cut = threshold × the bin's reference peak (peak-relative, so the knob means the same thing at any absolute level — but it is level-*history* dependent, not level-independent: see the reference below), then a per-bin one-pole across frames: instant attack, ~4-frame release. The gain is −6 dB at the threshold and falls as (mag/cut)² below it. Soft + smoothed, not a per-frame binary mask, because a hard mask makes bins sitting near the threshold blink at the hop rate ("musical noise" / birdies). Fully wet.

  The reference is per bin and held, and it has to answer two demands that a single number cannot:

  • a frame's own peak is self-referential — a quiet frame lowers its own bar, so the gate hands the floor back the moment the source stops. Measured on a −50 dBFS floor under a −6 dBFS tone, the floor was 63.7 dB down while the tone played and 3.0 dB down in the gap (spectralGate 0.5).
  • a globally held peak overshoots the other way — one loud note then gates the legitimately quiet phrase after it. Measured on a −34 dBFS phrase following a 0 dBFS note: −32.5 / −18.7 / −7.1 dB at 250 / 450 / 750 ms into the phrase (at the 0.35 of the example above, −26.5 / −13.6 / −4.4).

  Level history alone cannot separate them — in those two fixtures the residual sits 44 dB and 34 dB below the peak. What separates them is per bin, and it is a rise rather than a level: the floor in a gap was already sitting in its bins all through the loud passage, while a quiet phrase's partials arrive. So each bin carries its own reference. A bin that rises 12 dB above its own recent hold is new program and is credited as such; a credited bin tracks the program down fast (20 ms), an uncredited one keeps the 200 ms hold. Rising is instant either way, and the reference is stereo-linked (max(peak_L, peak_R)) like every other gate in the engine, so a hard-panned note cannot throw a centred tone's image. After the repair the gap still reads 36.6 dB down while the quiet phrase is penalised only 5.9 / 3.4 / 3.1 dB. Pinned by gate_stays_shut_when_the_source_stops, gate_re_references_a_quiet_phrase and gate_reference_is_stereo_linked.

• tilt — per-bin gain a · 6 dB/oct about a 1 kHz pivot, clamped ±24 dB per bin; fully wet.
• spectralMorph — the named sample is STFT-analysed once at build time into per-hop magnitude frames (control thread, like convolve kernels). Each live frame advances the sample cursor one hop (looping); output magnitude = lerp(live, sample × g, t) per bin with g a per-frame energy match (the sample's spectral shape at the track's level — the excitation keeps the dynamics), phases stay the track's. An unloaded sample is dry-through (browser patches the buffer in async, like granularize).

3. Determinism & tails#

Blur's phase randomisation is a per-channel xorshift32 with the fixed seed 0x5EC7 ^ channel, stepped once per bin per frame — a pure function of frame index, bit-identical on replay and offline render, matching on both backends. freeze at amount > 0 is an infinite tail; the insert-bypass reset clears the OLA buffers and drops the captured frame.

4. Implementation#

• Compiler — BuiltinKind::{Freeze, Blur, SpectralGate, Tilt, SpectralMorph} (builtins.rs), TrackEffect::{Freeze, Blur, SpectralGate, Tilt, SpectralMorph} (ir.rs), lowering arms in lower/track.rs, TrackRole::Effect in BuiltinKind::track_role (which is what both track-classifier guards read — they were hand-maintained lists until 2026-08-15); mixer inserts reuse that lowering (lower_section in compiler/src/lower/mixer.rs applies the insert to a synthetic track receiver). The parameter schemas live in compiler/src/builtins/track_effects.rs; spectralMorph's sample is a prepared string, so a non-string fails at check time (a type error).
• Engine — engine/src/spectral_fx.rs: the SpectralFx insert (realfft f32, allocation-free on the audio thread) + EffectState::Spectral build/process/reset/apply_macros arms in engine/src/effect.rs. analyze_morph_frames runs on the control thread at build. This is the one implementation everywhere — desktop, the browser AudioWorklet (engine-wasm/) and the plugin target. (The separate Web Audio worklet the browser used to run was deleted with the compatibility graph on 2026-07-24.)
• Tests — engine: the DSP tests in engine/src/spectral_fx.rs (identity at 0, freeze sustains + releases, gate kills quiet partials and re-references a quiet phrase, stereo-linked reference, tilt slopes both ways, morph endpoints, blur determinism) plus engine/tests/spectral_e2e.rs; compiler: lowering + clamp + Str-arg tests in compiler/src/lower/tests.rs; golden IR snapshot for examples/songs/hoarfrost.volta.

5. Deferred#

See Spectral effects §7 — per-voice two-signal morph, spectralMorph (tap …), an explicit freeze re-capture trigger, bus latency compensation, a bigger-FFT blurDeep.