Analyze and rebuild a sound

Separate a recording into sinusoidal components and residual sound, then reshape the analyzed material during resynthesis.

partialsOf (load-time SMS analysis) and partials (the playable source), plus the pTime / pPos / pNoise / pStretch / pTilt / pMorph / residualOf / pFormants pipe family. Design spec: Analyze and rebuild a sound. Demo song: examples/songs/loom.volta.

The tier above the phase vocoder. A recording is taken apart into tracked partials — per-frame frequency and amplitude trajectories matched frame-to-frame by McAulay–Quatieri birth–death — plus a stochastic residual, and the analysis becomes a playable, transformable value. Because the sinusoids are parameters rather than spectra, stretching is moving a cursor (no smear, no phasiness, cost independent of the ratio), repitching is one multiply per track (no formant chipmunking), time is addressable by a signal, and two sounds can be crossed at the skeleton/breath seam. This is Serra & Smith's SMS model, and the pitch-tracking capability Wavetable From Sample §8 explicitly reserved for "a different feature".

1. Surface#

-- Reusable analysis values.
cello  = partialsOf "cello-a2"
breath = partialsOf (sample "flute") (maxPartials 96)
someBounce = bounceOf 4c (play bass bassPat)
tape   = partialsOf someBounce                 -- a bounceOf/sampleOf/comp handle

-- A playable source, repitched by the tracking ratio (analysed f0 → played hz).
solo   = patch { source = partials cello (freq pitch), gain = vel * 0.8 }

-- Transforms are pipe modifiers that FOLD INTO the node.
slowed  = patch { source = partials cello (freq pitch) |> pTime 24 }
scrub   = patch { source = partials cello (freq pitch) |> pPos (lfoTri @ 8c |> scale [0, 1]) }
airless = patch { source = partials cello (freq pitch) |> pNoise 0 }
hybrid  = patch { source = partials cello (freq pitch) |> residualOf breath }
grounded = patch { source = partials cello (freq (pitch - 12Semitones)) |> pFormants }
snapped = patch { source = partials cello (freq pitch) |> tuned 0.8 }

partialsOf returns a PartialsAnalysis value. Use it inline, bind it locally or at top level, return it from a function, import it, or store it in a list, tuple or record. partials, pMorph and residualOf accept the same value. An analysis is not a sample buffer or a string: sample analysis and partials "name" 220Hz are type errors.

config : PartialsConfig = maxPartials 32
analyze = \name -> partialsOf name config
sources = [analyze "sax"]
p = patch { source = partials (head sources) (freq pitch), gain = 0.1 }
t = play p !"c4"

The source and partial count are fixed during compilation; a live macro cannot select an analysis configuration. maxPartials takes a finite dimensionless count of at least 1, rounds it, and clamps it to 8–256. Reusing or reconstructing the same source/count request shares the prepared analysis. Each sounding note still owns its playback cursor and phases. Naming or importing the value does not change the resource key.

Live timbre controls#

Open Prism Current in the song chooser for a complete, portable performance (examples/songs/prism-current.volta). Hear its original reed, then move prism, color, air and scan. All source audio is synthesized locally.

• pNoise controls the residual level independently of the pitched partials. Zero removes the residual; it does not mute the pitched sound. A live zero keeps the noise filters advancing so restoring the level cannot resume stale filter memory.
• pStretch raises each partial-to-fundamental frequency ratio to an exponent. At 1 the frequencies are unchanged. Above 1 the upper components spread apart; below 1 they bunch together. The fundamental remains anchored. Existing tuned frequency correction is applied before the exponent.
• pTilt applies a slope in dB per octave relative to the analyzed fundamental. Use a plain number: pTilt (-6) attenuates a partial one octave above the fundamental by 6 dB, and one two octaves above by 12 dB. Positive slopes brighten. Each partial's tilt gain is bounded to ±24 dB; leave headroom when boosting. Residual bands are unaffected.

All three accept constants, macros and dimensionless control signals; physical units and audio-rate signals are rejected. Live values are read at block endpoints and interpolated through the renderer's frequency/amplitude ramps. Phase accumulators continue when timbre changes. Out-of-range controls clamp; nonfinite live values fall back to the compiled initial value. Components fade out in the top 15% of the Nyquist band. Neutral constants omit the extra IR fields and preserve existing renders. A later modifier replaces the earlier constant or signal for the same control.

Morph between analyzed sounds#

reed = partialsOf "reed"
glass = partialsOf "glass"
blend = macro 0
p = patch {
  source = partials reed (freq pitch) |> pMorph glass blend
    |> pPos (const 0.2) |> pTilt (-3) |> pNoise 0.2,
  ampEnv = adsr 80ms 0ms 1 250ms, gain = vel * 0.15
}
t = play p "c3:min"

pMorph changes a sounding note between two analyses. At 0 it uses the primary source; at 1 it reaches the destination's partial ratios and amplitudes. Matched frequencies interpolate geometrically, so their pitches move between the endpoints. Amplitudes interpolate linearly. Unmatched components retain their own frequency as they fade. The destination is repitched relative to its own estimated fundamental, while the note stays at the played pitch.

The primary source owns duration. pTime advances both analyses through the same normalized position; pPos holds or scrubs both. Different recording lengths therefore fit the same note. Residual-band envelopes blend with the morph amount, unless residualOf explicitly supplies the noise source. Stretch, tilt, formant preservation and tuning also apply. Matched oscillator phases continue when the control moves; an endpoint is a resynthesized spectrum, not a phase-identical copy of the original recording.

Both analyses and their correspondence must be prepared before the voice sounds. Preparation is shared per source pair. It greedily pairs overlapping normalized track lifetimes by closest fundamental-relative mean pitch, up to 600 cents apart, with deterministic tie breaks. It preserves the primary track order and appends unmatched destination tracks. Two 256-track analyses require at most 512 oscillator lanes. This preparation never runs in the voice renderer. A voice reused for a different patch binds that patch's analyses before sounding; a missing or unusable analysis cannot replay the previous patch's bank. Replacing a recording rebuilds its analysis and prepared pairs for new notes; an already sounding voice retains its analysis snapshot.

Use related, pitched material for smooth results. This rule does not align syllables, transients or semantic features; imperfect fundamental estimates also affect the result. A silent destination fades the primary sound out. A constant zero removes the morph dependency and retains the ordinary render path. Live zero retains the prepared lanes, ready to move on a held note.

Try Glass Orchard (examples/songs/glass-orchard.volta) for a source-pair introduction and evolving chords, or Tidal Alloy (examples/songs/tidal-alloy.volta) for a morph feeding rhythmic grain capture. Both synthesize their own source audio locally.

Partials inspector#

Open the source's inline viewer in the notebook. Its partials controls appear below the audition spectrum on the shared native/browser workbench. Drag Morph, Stretch, Tilt or Residual to edit literal arguments. A morph destination is chosen in source with pMorph target amount; the Morph control becomes available when that stage exists. Hold… adds a position control; Resume… restores the time clock. The last authored clock or repeated modifier owns the value.

Named literal arguments are editable too. An expression appears as Expression ↗ and navigates to its source, preserving authored macros and modulation. A drag is one undo operation, and guarded source splices reject a changed document. Parsed bindings are cached by exact text and selected source; repainting does not reparse. Controls wrap within the visible viewport. The existing audition spectrum displays measured preview audio and says when an audition is needed; it does not invent live analysis telemetry.

2. Analysis (control side, deterministic)#

engine/src/partials.rs::analyse_partials, run only on the control thread by EngineHandle::bake_partials — thousands of FFTs plus track matching, which is orders of magnitude past the audio callback's deadline. One single pass over SampleData.mono (the never-changing analysis view):

1. Trim leading/trailing near-silence so frame 0 sits on the attack; cap at 30 s (a longer source is analysed up to the cap and the bake driver prints volta: partialsOf "x" is longer than 30s …).
2. STFT — FFT 2048, hop 256 (8× overlap), Blackman–Harris window, f32 via realfft. 5.33 ms frames at 48 kHz.
3. Peak picking — 5-point local maxima above an absolute −80 dBFS floor and a relative peak threshold, refined by parabolic interpolation on the log-magnitude (sub-bin frequency and amplitude). Capped at 4 × maxPartials peaks per frame so a noise burst can't make the matcher quadratic.
4. MQ birth–death matching — each live track claims the nearest unclaimed peak within ± max(0.03·f, 30 Hz). Unmatched tracks continue virtually for up to four frames with decaying amplitude before retiring; unmatched peaks are births. Tracks shorter than 3 frames are dropped as tracker noise. The strongest maxPartials tracks by total energy survive, each carrying start (birth frame), per-frame (freq, amp) as f32, the measured birth phase, and an amplitude-weighted mean_hz.
5. f0 — onset::estimate_root_hz (the engine's McLeod/NSDF tracker, median of per-hop estimates). Unpitched material falls back to the lowest surviving track's mean, which is the right answer for a bell or a plate. f0 = 0 means unusable and the voice stays silent.
6. Residual — a copy of the frame's magnitudes with every peak's region of influence (±4 bins for the Blackman–Harris main lobe) zeroed, integrated into 24 Bark-ish band RMS values. No inverse FFT: resynthesis is seeded filtered noise shaped by those envelopes (Serra's stochastic model), which is what makes the residual stretch-friendly and deterministic.
7. Spectral envelope — a two-pass box smoothing of the log-magnitude (the vocal.rs estimator), sampled at the same 24 band centres. Only pFormants reads it.

Determinism is a property of the algorithm, not of a seed: no RNG, and the final track sort breaks energy ties on mean frequency, so two bakes of one buffer are bit-identical (pinned by engine/tests/partials.rs).

Storage: a SampleBank side table keyed (sample name, maxPartials) — the resynth/modal plan, so the audio-thread lookup borrows and never allocates. Memory for 30 s at 64 tracks is ≈3 MB.

3. Render (audio thread, allocation-free)#

CompiledNode::PartialsSource → NodeState::Partials → partials::PartialsRender.

• The latch. The node renders SILENCE and retries the bank lookup every block until the bake arrives. This is the AdditiveResynth discipline, not the Stretch pre-bind — which is the difference between "sounds as soon as the sample lands" and "silent forever because the sample arrived after the module" (the bounce-stretch war story). bake_partials is called from both load_module and load_samples, so whichever arrives second completes it, and it is idempotent per key. What a note bound — source, residual and morph plan together — sits in a NoteBound keyed by the voice's note serial (engine/src/voice/state.rs), so a reused slot cannot read an earlier note's bind even if a reset forgot it. A replaced recording's analyses, prepared pairs and "already shipped" marks are dropped by the bank itself on every install path (SampleBank::forget_derived).
• Sizing. Per-track phase accumulators, per-track block-ramp scratch, the tuned multipliers and the 24 residual bandpasses are all sized once at latch and reused in place across retriggers (reset_node_state re-arms rather than dropping to None). engine/tests/rt_safety.rs carries a partials scene that fails if a note-on ever re-allocates.
• Per block, each track's (freq, amp) is sampled at the block's first and last sample and linearly ramped between — finer than the 5.33 ms analysis grid, so nothing steps. Per sample it is one sin and one phase increment per track. Each analysis is capped at 256 tracks; a morph can use up to 512 lanes.
• Repitch is hz / f0 applied to every track's frequency. Partials above Nyquist drop out through a fade over the top 15 % of the band.
• Births under stretch keep their attack alignment by construction: the cursor is measured in analysis frames, so a track born at frame 40 sounds when the cursor reaches frame 40 — at 1× that is 213 ms in, at pTime 24 it is 5.1 s in, and the internal shape of the attack is preserved either way.
• The end is the end. On the internal clock, once the cursor runs past the last frame both the tracks (already zero) and the residual stop. A re-articulated note on a glide voice (which is persistent: every note after the first is a renote of the same voice) restarts the clock at frame 0, with the track phases running on so the re-attack has no seam; a legato (slid / overlapping) note keeps the cursor, as a sample keeps its playhead. Under pPos the curves are held instead: a scrub parked at 1.0 is addressing the last frame, not falling off the end of one.
• Residual — 24 TPT state-variable bandpasses at the Bark centres, Q = fc / bandwidth, fed by one shared noise stream. Each band's gain converts the analysed band RMS into filter input using the 2-pole bandpass's equivalent noise bandwidth ((π/2)·fc/Q), so pNoise 1 reproduces the analysed level rather than an arbitrary hiss, and pNoise scales it exactly linearly.
• Noise determinism — the LCG is re-seeded from a splitmix32 of the analysis frame index at every frame crossing, so the residual is a pure function of where in the analysis you are: a scrub back over the same ground hears the same texture, and two renders are bit-identical.
• Mono by construction. A bank of sine oscillators has no width to carry (the AdditiveResynth precedent, Stereo Samples); carries_stereo is false. Use width/jux/pan downstream.

pFormants#

At render, each track's amplitude is multiplied by env(f_played) / env(f_analysed), where env is the analysed spectral envelope of the current frame interpolated in log-frequency. That is exactly "divide the envelope out and re-apply it at the played frequency": a voice transposed down keeps its formants where they were, so it reads as a low voice rather than as slowed-down tape. Measured in engine/tests/partials.rs — the dominant peak stays within ±320 Hz of a synthetic 1500 Hz formant under ±7 semitones, and always closer to it than the un-preserved render.

residualOf#

A's tracks over B's noise bands. The two analyses keep independent frame cursors driven from the same normalized position, so pTime and pPos stretch and scrub both together. Both bakes must land before the voice sounds (a half-formed hybrid is worse than a beat of silence), and bake_partials collects the partner's (sample, maxPartials) alongside the primary's.

tuned on partials#

The tracks do not exist at compile time, so there is no ratio table to rewrite the way Match timbre to tuning does for additive/additiveOf. Instead the node carries the resolved cents lattice and the engine snaps at latch: each track's mean ratio-to-f0 is snapped (partials::snap_ratio_cents, the engine-side twin of compiler/src/tuning.rs::snap_ratio) and the whole trajectory is shifted by that one constant factor. Vibrato, glissandi and the attack's frequency sweep therefore survive intact — only the intonation moves. With no tuning = … declaration the lattice is empty, the snap is the identity, and the usual TunedNoTuning warning fires.

4. pTime ⇄ pPos#

Both decide the frame cursor, so they are mutually exclusive: the last one in the pipe wins, and the loser produces a PartialsClock warning diagnostic rather than being silently dropped.

pos = lfoTri @ 8c |> scale [0, 1]
-- pPos wins; the pTime 4 is reported as ignored.
posWins  = partials cello (freq pitch) |> pTime 4 |> pPos pos
-- pTime wins; the position signal is reported as ignored.
timeWins = partials cello (freq pitch) |> pPos pos |> pTime 4

5. IR & seams#

// Module
"partialsDefs": [{ "name": "cello", "source": "cello-a2", "maxPartials": 96 }]

// A node (defaults are omitted from JSON, so nothing churns)
{ "op": "PartialsSource", "freq": "n1", "sample": "cello-a2", "maxPartials": 96,
  "timeRatio": 24.0, "pos": "n11", "noise": 0.7,
  "residualSample": "breath", "residualMax": 12, "formants": true,
  "tunedCents": [...], "tunedAmount": 0.8 }

The node carries the resolved source key and count (not the handle), which is what lets bake_partials scan compiled nodes exactly like bake_resynth does, with no module-level def list on the audio side.

Offline renders (export, bounceOf, render_track_mono) drive an AudioEngine directly — no command ring — so they call AudioEngine::bake_partials_offline before load. Without it a partials voice would be silent in every export, which is the same class of bug as bounce-stretch.

6. Choosing maxPartials#

Tracks are kept strongest-first, so the count is a deliberate simplification knob, not just a budget:

Render cost is ≈0.4 × maxPartials + 8 in the cost model's baseline-oscillator units, and it does not depend on pTime.

7. Limits & deferred#

• 30 s cap. The bake runs on the worklet thread between quanta in the browser; a longer source is truncated with a printed warning. Chunked bakes across quanta are the follow-on.
• Peaks that don't survive the top-n cut are removed from the residual (the residual is computed in the same single pass, before the energy sort), so a very low maxPartials on dense material loses a little energy. This is the price of a single-pass, bounded analysis; it is inaudible at the default.
• Mono only. No mid/side analysis (the resynth precedent).
• No live-input analysis. Real-time partial tracking is a different budget class; deferred.
• No disk cache. The bake is in-memory-idempotent per (name, count); a ~/.volta/partials cache keyed like StretchKey is the follow-on.
• Per-partial user transforms (pMap (\i f a -> …)) — harvested to Ideas.

8. See also#

• Additive synthesis — additiveOf, the cheap harmonic ancestor.
• Time-stretching and pitch-shifting §7/§8 — the phase-vocoder smear this replaces.
• Match timbre to tuning — the shared snap_ratio kernel.
• Strings, bells and resonating objects — modalOf, the other bake-an-analysis source.
• Vowels, voice and pitch effects — the spectral-envelope estimator pFormants reuses.