Neural sound transformation

Explore model-based audio transformation and generation, with the required model packages and supported host paths made explicit.

Full encode → latent controls → decode runs through tract in Studio on desktop and browser, without PyTorch or libtorch at playback time. No default build includes it — the neural cargo feature is opt-in everywhere, and the default release ships without it. Studio opts in with cargo build -p volta-studio --features neural (without it, a model import answers "This build has no neural model support"). The browser engine opts in with ./build-engine-wasm.sh --features neural (a default build refuses neural songs and package imports with "Neural models are not included in this browser build"). The CLI opts in with cargo build -p volta-cli --features neural (a |> neural song still compiles without it — only rendering needs the backend). A versioned split ONNX package supplies the encoder, decoder, streaming buffers and noise contract. Direct TorchScript playback and the optional neural-torch feature were removed on 2026-09-08; convert supported archives before importing them. Removal evidence: Neural Tract Only 2026 09 08. Design: Rave Tract, Rave Model Library, Neural Instrument, Latent Timbre.

Try it in Studio#

examples/songs/rave_tract.volta, RAVE — Skin into Metal, is registered in the song library. Its 24-bar arrangement first plays the two encoder inputs, then demonstrates resynthesis/morphing, freeze and decoder generation. morph blends the two encoded timbres; warp, edge and air offset latent lanes; freeze holds the current timbre. neuralLevel and dreamLevel set the two neural voices' levels. Keep skinT and metalT running: they feed the encoders. Scene changes apply their own macro settings; adjust the controls after a scene starts to override its settings.

Library → Samples → Models → Import model… accepts a prepared .rave file in both Studio hosts. The list shows each model's latent width and sample rate. Insert instrument writes an editable encoder-input track and a decode/offset/smoothing/hold instrument with one macro per exposed lane. Place it at the top level in Source and run both tracks. Copy instrument provides the same snippet on the clipboard. Import checks both ONNX graphs and runs a warmup before reporting success; a malformed package does not replace the song.

examples/songs/rave_sol.volta, RAVE — SOL / Eight Windows, provides the 24-bar tour for SOL with all eight latent lanes exposed. It uses one encoder and one decoder: morph moves from encoded percussion toward a seeded latent prior, and dreamLevel sets that prior's motion. Its lane labels are audition prompts, not guaranteed learned meanings. Import sol.rave and open it from the song picker. Both demos use locally supplied model weights.

Create a portable package from a checked export:

python3 scripts/install-rave.py /path/to/sol-export --name sol \
  --package /path/to/sol.rave --package-only

Alternatively, install into both the desktop directory and a local browser checkout without opening the picker:

python3 scripts/install-rave.py /path/to/rave-export --name darbouka
cargo run -p volta-studio --bin volta-studio -- examples/songs/rave_tract.volta

The installer writes ~/Documents/Volta/models/darbouka.rave.json and darbouka/{encode,decode}.onnx, plus the same assets under runtime/public/models/. It refuses to replace different existing bytes. Browser assets and weights are gitignored. Browser imports persist in IndexedDB for the current browser profile and site; clearing site data removes them. Weights are not uploaded or included in shared songs. A desktop installation does not install them on a remote site. Rebuild the engine (with the otherwise-off neural feature) and the UI for a local browser checkout:

./build-engine-wasm.sh --features neural
./build-ui-wasm.sh
cd runtime
npm ci
npm run dev

Conversion is separate from playback#

The checked exporter is spikes/rave-tract/export.py; its README gives the Python environment and command. It reconstructs supported mono variational v1/v2 deployments from archive weights and attributes, exposes cached-convolution buffers and random draws as graph inputs, and checks against TorchScript. PyTorch is only needed for that export/reference step. The resulting ONNX pair runs in Rust on either platform.

Validated archives are Darbouka (v1, four exposed lanes, 55 streaming buffers) and SOL (v2, eight exposed lanes, 58 buffers, amplitude modulation enabled). Both use 128 internal lanes, 16 PQMF bands, a 2,048-sample hop and 44.1 kHz. The inspected SOL archive has AdaIN disabled. The converter reads dimensions and sample rate, measures the actual one-frame decode length, then compares streaming encode/decode and edited-latent reference fixtures including resets. Older archives may declare sample rate in sibling index.json; otherwise pass --sample-rate. --hop, when supplied, must match the measured length.

Other archives using these supported blocks are conversion candidates, not automatically validated models. Active AdaIN, stereo/multichannel audio, discrete latents, noise-synthesis heads, non-PQMF modes, resampling wrappers and unfamiliar blocks are rejected. Exposed width must be 1–16 and smaller than the internal latent width. Performance must be measured per model and device. Earlier evidence: Rave Tract 2026 09 07, Rave Studio 2026 09 07.

SOL is substantially larger: its portable package is about 120.5 MiB versus Darbouka's 18.6 MiB. The SOL demo deliberately uses one encoder and one decoder. Two encoders plus two decoders exceeded the measured browser budget on the validation Mac; numerical compatibility does not establish that every combination will run live. Offline rendering supports heavier combinations. Keep the device's load and neural deadline statistics in view when adding units.

Chromium 149 passed a minute of live playback for each demo with zero missed inference hops. SOL's inference service p95 was 22.5 ms; Darbouka's was 10.3 ms for its two-unit batch. These are local measurements against a 46.44 ms hop budget, not device guarantees or end-to-end latency measurements. Firefox 151 imported and reopened SOL, but its 239 ms service p95 missed every live deadline on the validation Mac; its neural output was silent. Playwright WebKit 26.5 produced SOL audio with a 19 ms service p95, but missed twelve hops in its minute run, so it also failed the zero-miss acceptance check. Use Chromium for the validated browser workflow. Full evidence: Rave Model Library 2026 09 07.

Backends and model resolution#

Native lookup tries the supplied path, then .rave.json and .onnx; it repeats that order under ~/Documents/Volta/models/ and ~/.volta/models/. Successful loads and failures are cached for the process. Restart Studio after replacing a model in place. Explicit .ts paths report conversion guidance; they are never loaded through a hidden fallback. Missing native inserts pass dry through with a warning; missing decode units produce silence.

Browser Studio first resolves locally imported packages, then asset names (letters, digits, _, -) under /models/<name>.rave.json. It validates both graphs in workers before accepting the new song. Missing or invalid packages report a load error and preserve the previous installed song. It cannot load arbitrary filesystem paths or TorchScript archives.

Desktop imports publish graphs first and the manifest last using atomic, non-replacing file creation; imported plans immediately bypass a cached load failure. Browser metadata and package bytes commit in one IndexedDB transaction. Repeated identical imports are harmless; a different package with an existing local ID is refused. A browser model already loaded from server assets also reserves its ID for that session. Export under a new ID to compare variants. The setup script writes /models/catalog.json listing staged browser models (sync-assets writes [] if none); local imports need no catalog.

Split package contract#

A portable .rave file is a ZIP containing exactly model.json, encode.onnx and decode.onnx. model.json contains {name, manifest}; manifest follows the schema below with the two graph basenames. IDs contain 1–80 ASCII letters, digits, underscores or hyphens. Stored and deflated ZIP entries are supported. Imports cap the package and its expanded contents at 256 MiB, and metadata at 1 MiB. Duplicate, extra and escaping entries are rejected. The importer does not execute Python or arbitrary archive code.

format: "volta-rave-split", version: 1 declares sample_rate, hop, dims, optional seed (default 0x5eed), and models.encode/models.decode. Each graph declares its relative filename, ordered input/output names, input, noise and output shapes, and the shapes of all cached states. Port ordering, float32 dtype and exact shapes are checked against ONNX before optimization.

The mono encoder maps [1,1,hop] to [1,dims,1]; the decoder reverses this. Noise and previous states are explicit inputs; next states follow the primary output. Supported manifests have 1–16 exposed lanes and a fixed hop. Darbouka includes PQMF, variational sampling, PCA/inverse PCA and the decoder's 124 residual noise lanes, with all 55 streaming buffers retained. Each tapped source owns an independent encoder history; a unit owns its decoder history.

Split packages declare their sample rate directly. Legacy forward-only ONNX can supply index.json beside its model ({"model_name":{"sr":44100}}); without it the engine assumes its own rate. The engine converts between model and device rates at hop boundaries using its existing Catmull–Rom resampler. A 44.1 kHz model therefore keeps its training rate on a 48 kHz device.

Inserts and execution#

tex = play pad mel |> neural "darbouka"       -- mix defaults to 1
blend = macro 0.6
duo = play pad mel |> neural "darbouka" blend

neural is a per-track insert, chainable with the other effects. Mix is clamped to [0,1]. neuralMod adds per-lane biases (below). A mono model reads (L+R)/2; its wet output is duplicated to stereo.

• Native: each insert/unit has an inference thread with ordered hop exchange. The audio callback receives the preceding hop before dispatching the next. It can block if the model exceeds its budget.
• Browser Studio: an AudioWorklet exchanges hops directly with a dedicated Worker over a MessageChannel. Rust still evaluates the complete latent program. The worklet never waits: an overdue hop becomes zero wet PCM. There is at most one outstanding inference hop per unit; late replies are discarded, reset invalidates old sequences, and recovery resets streaming state before accepting fresh input. neural_stats() reports completed and missed hops. Serialization and buffering allocate; this is not an allocation-free real-time design.
• Offline: desktop renders use the ordered native exchange. Browser bounces, captures and exports execute synchronously in a separate Worker, with mirrored sample/wavetable assets, so an export does not occupy the live inference worker. The same Rust graph and latent evaluator render both. A native offline render never prints a neural part as silence: the Studio exporter, MCP measurement, volta export stems and the render-manifest example all call CompiledModule::require_neural_models, which refuses the render by model name when the build has no neural feature (the CLI and engine examples are built without it unless asked: --features neural), or when a neural VOICE's model does not resolve. A missing INSERT model is allowed with the runtime present: it passes dry offline exactly as live playback on that machine does.
• Latency: two engine-rate hops are prefilled with silence (about 93 ms for Darbouka), plus the model's own streaming response. Decode units feed this declared lookahead into track delay compensation. Insert dry remains undelayed, as with the existing lookahead-effect convention.
• Reproducibility: split backends own separate seeded encoder/decoder Gaussian streams and all caches. Reset restores both. Repeated offline exports are sample-identical on the same runtime/machine; native/Wasm floating-point results are compared with tolerances. PyTorch uses a different RNG, so reference parity supplies the same explicit noise inputs. Live browser deadline misses can change playback.

The workspace pins tract 0.21.10 with a documented alignment fix in vendor/tract-linalg/. Unpatched generic binary kernels assert on legal misaligned decoder tensor views in Wasm. The engine build enables Wasm SIMD. See vendor/README.md for provenance and the dependency-version constraint.

Latent controls#

The latent is a control-rate vector, one frame per model hop (about 21.5 Hz for Darbouka). Macros, LFOs and patterns supply its lanes and control leaves. Use split packages; legacy forward-only ONNX cannot implement these operations.

Insert with biases — track |> neuralMod "model" [bias…] [mix]. Per-dim additive offsets on the latent (z' = z + bias), each slot a live Param (macro/LFO/pattern like mix). A shorter list leaves the tail un-biased. neural is exactly neuralMod m [] mix. dim 0 is the most informative lane (PCA order), so a macro on it is a real timbre knob.

A model generates on silent input, so both inserts are gated on their track's activity: once the track stops or leaves the scene the wet stream fades to silence over 0.2 s (the dry path is untouched), and it returns at once when the track plays again (The effects reference § "What happens when a track stops").

warp = macro "warp" 0 (lfoSine 0.08Hz * 1.5)
duo  = play pad mel |> neuralMod "sol" [warp, 0, -0.4] 0.8

The latent as a value. encode "model" (tap "track") → Latent and decode "model" <latent> → audio split the box; combinators shape the latent between them — latentSmooth z ms (per-lane slew), latentHold z gate (freeze = held-timbre drone), latentQuant z step, latentAdd z [offsets…] (the value form of a bias vector), and latentXfade zA zB t (timbre morph between two sources). One model per unit (a mismatched inner encode is a compile error). encode used standalone errors — it produces a latent, feed it to decode. Its source names a declared track, a bus or "input", like every other cross-track reader, and a name that matches none of them is refused before installation rather than decoded as silence (validate_for, Language Foundations P4).

A transient latent is an ordinary value, like a capture: name it, import it, return it from a function, hold it in a record or a list, bind it with let — the walk asks the value, not the spelling. shaped = latentSmooth cap 40ms is a latent binding, not a synth, and nothing else can consume one. Anything that is not a latent is refused at the receiver with one sentence (expected a latent — …) whatever context carried it there. A capture and a transient latent are still different TYPES, so a conditional cannot choose between one of each; make both arms latents.

strings = play pad mel                     -- the track the `encode` taps
voice = patch {
  source = decode "darbouka" (
    latentAdd (latentSmooth (encode "darbouka" (tap "strings")) 40ms)
      [slowLfo, 0, 0, wob]
  ),
  gain = vel
}

Generation — the decoder as a source. neuralOsc "model" [laneSig…] runs the decoder with no input audio over hand-authored latent lanes (LFOs / envelopes / patterns), played like a modular. neuralPrior "model" temp [seed] walks a seeded latent trajectory (leaky-integrator noise; seed defaults to the call's source offset, so it is replay-stable).

tex = patch { source = neuralOsc "sol" [lfoSine 0.05Hz * 2, slowLfo, 0, wob], gain = vel }

Stored coordinates — latentOf and the navigation vocabulary. Everything above is transient: a latent exists for one model hop. latentOf makes one persistent, and the rest of the family navigates the space between captures (see Latent Timbre).

celloZ = latentOf "sol_full" (sample "cello-a2")   -- the binding IS the coordinate
fluteZ = latentOf "sol_full" (sample "flute-c4")
brassZ = latentOf "sol_full" (sample "brass-e3")

morphK = macro "morph" 0.3
tourK  = macro "tour" 0

bed   = decode "sol_full" (latentXfade celloZ fluteZ morphK)
tour  = decode "sol_full" (latentPath [celloZ, fluteZ, brassZ] tourK)
darker= decode "sol_full" (latentAxis celloZ fluteZ 0.3 celloZ)
ghost = decode "sol_full" (latentPlay celloZ 0.5)
scrub = decode "sol_full" (latentScrub celloZ (lfoTri @ 8c |> scale [0, 1]))

Captures pass through aliases, functions, imports, local bindings, records, lists and tuples. Model names and source keys resolve statically; each receiver keeps the source/model pair. Named metadata is for inspection, and inline captures prepare without a declaration catalogue. A capture is neither PCM nor a string: sample, sampleAt and take reject it; latentPlay and latentScrub reject strings and uncaptured transient latents. Replace an old Str annotation on a capture with LatentCapture.

makeCapture = \model source -> latentOf model source
stored = { flute = makeCapture "sol_full" "flute-c4" }
voice = patch { source = decode "sol_full" (latentPlay stored.flute 0.5) }

• Capture (engine/src/neural/capture.rs) resamples the sample's mono view to the model's rate and drives a synchronous instantiate_mono() backend hop by hop — control-side, never the audio thread and never the per-hop worker, so replay adds no encoder cost at play time; decoding still runs. Sources longer than 60 s are truncated (head kept) with a warning; models wider than MAX_DIMS = 16 truncate to 16 lanes, exactly like neuralMod biases.
• Determinism. Split captures use private seeded noise and fresh streaming buffers. Repeated bakes agree on the same runtime and machine.
• The bake seam. Live: EngineHandle::bake_latents, the exact bake_resynth twin — idempotent per (sample, model), driven from both load_module and load_samples so whichever arrives second completes it. Offline: AudioEngine::bake_latents_offline, the fourth arm of bake_analyses_offline, so export / bounceOf / freeze bake it too. The capture reaches the audio thread as Command::InsertLatent and is handed to the decode units that read it — silence until latch: before it lands, a coordinate supplies zero latent lanes. A real decoder can generate audio at those coordinates; zero latent does not necessarily mean silent PCM.
• Typed endpoints. A coordinate's model must equal the enclosing decode's (celloZ under decode "other" is a compile error). A coordinate is meaningless in another model's space, and this is the claim the feature rests on, so it is checked rather than assumed.
• Source identity. Capture requests use the source bank key and model, never the authored binding name. sample celloZ is a type error; the PCM fetched for encoding is the capture value's source.
• Free-running playback. A decode unit is shared across the voices that reference it, so latentPlay is anchored to the transport, not to note-on. Per-voice coordinates would need a per-voice unit pool — a different budget class, deferred.
• Latency. A decode unit pre-fills its stream with 2·eng_chunk zeros (one hop accumulating, one in flight) — the same lookahead the neural insert reports. It now feeds the engine's PDC sum, so a morph bed sits on the grid with the dry tracks beside it.

Engine: each decode/neuralOsc/neuralPrior occurrence compiles to a NodeKind::NeuralDecode over a LatentIr tree (ir/src/ir.rs); the engine builds one persistent mono NeuralUnit per occurrence (engine/src/neural/unit.rs), pumped once per block from the previous block's buses (the tap discipline) before the voices render. A unit is shared across the voices that reference it (a single drone — last control write per block wins).

Transient encode/combinator expressions must be nested inside decode; a standalone top-level latent binding currently fails lowering. Persistent latentOf captures are ordinary values. encode taps tracks/buses; live microphone encoding through the latent-unit path is not implemented.

Native sample index#

Studio's sample drawer can build its "sounds like" index with a split package in a desktop build with neural. The first loadable split package in the model directories supplies the encoder. Browser library indexing remains unavailable.

Each sample contributes per-lane mean and population standard deviation. Similarity is cosine over the combined normalized vector; nonpositive results and the sample itself are excluded. Captures run on the index worker, with incremental progress. ~/.volta/latent-index.json caches entries. Its existing runtime_version field carries the tract runtime stamp; older caches with the torch_version field still load when their tract stamp matches. Caches built by the retired TorchScript backend are invalidated. Validation hashes the split manifest and both graphs, so changing any model bytes invalidates the index. The index is not a model picker.

Features and tests#

The engine crate keeps neural optional, and no product enables it by default. volta-studio once did; on in its defaults, it unified the model runtime into every cargo test --workspace binary, until the debug volta-ui test suite outgrew the __eh_frame macOS ld can index (app-native/Cargo.toml). volta-cli stays off too (the feature adds ~80 crates and the vendored tract-linalg asm build to every CLI build, which CI and the lesson validators run on each push; see cli/Cargo.toml), and neither does volta-engine-wasm (the same crates roughly double the worklet's dependency graph and its .wasm size; engine-wasm/Cargo.toml). The browser build opts in with ./build-engine-wasm.sh --features neural. A build without it exports the same neural names as stubs that refuse ("no neural inference"; engine-wasm/src/lib.rs, checked by runtime/src/engine-neural-surface.test.ts), so nested-song bakes in the offline worker keep working, neural_enabled answers false and the model library panel says the build has no neural. The browser harness below skips (or fails under --required) when the staged worklet lacks the feature. No libtorch dylibs are needed for split packages, including packaged desktop builds. The workspace has no tch or torch-sys dependency and no optional libtorch build. PyTorch remains only in the separate Python export/reference environment. make test-neural runs all six fixture suites below without Python or external model downloads.

engine/tests/neural.rs covers forward inserts and fallback; neural_package.rs covers portable imports, malformed archives and overwrite refusal; neural_unit.rs covers the latent laws; neural_split.rs covers schema/port validation, explicit state/noise, independent encoder histories, full voice rendering and bounded browser transport. latent_capture.rs and latent_index.rs run against tiny split fixtures, without PyTorch. The vendor alignment regression tests all relative offsets and short tails.

# The six files are modules of the engine's single `suite` harness
# (engine/tests/suite.rs), gated on the feature — select them by module path.
cargo test -p volta-engine --features neural --test suite -- \
  neural:: neural_unit:: neural_split:: neural_package:: latent_capture:: latent_index::
VOLTA_RAVE_ARTIFACTS=/path/to/rave-export cargo test -p volta-engine \
  --features neural --test suite neural_split::real_rave_encode_decode_matches_export \
  -- --ignored --nocapture
cd runtime
node scripts/studio-neural-browser-test.mjs --required
VOLTA_RAVE_MODEL=sol VOLTA_RAVE_PACKAGE=/path/to/sol.rave \
  VOLTA_RAVE_SECONDS=60 node scripts/studio-neural-browser-test.mjs --required

The real-model check compares the exported encoder/decoder and edited-latent sequences with reference noise and verifies that the full demo arrangement includes neural audio. Export --hops 128 or longer to set the fixture length. The browser check uses real Studio, live bus taps, macro changes and repeatable worker exports. Supplying a portable package also checks file-picker import, persistence across reload and rejection of a damaged package. Set VOLTA_RAVE_BROWSER=firefox or webkit to select another installed Playwright engine. These are automated measurements; they do not substitute for listening. Stock RAVE offers timbral control, not guaranteed note-conditioned pitch synthesis.