Generate samples with Magenta RealTime 2

Design: Generate samples with Magenta RealTime 2 and Magenta Instruments. Create sounds locally in Volta Studio on an Apple Silicon Mac, then use the results as ordinary samples in native or browser songs.

Play a text-driven instrument#

Describe the sound, then give it an ordinary note or chord pattern:

tempo = 120bpm
keys = magenta "soft electric piano"
t = play keys "[c4,eb4,g4] [d4,f4,a4]" |> reverb 0.35

No input recording is needed. magenta is a complete instrument source: one continuous model runs for each track that plays it. Track effects, mixer faders, gainTrack and panTrack process its stereo output. Notes guide the model's pitches and attacks, including chord notes, repeated attacks, releases, rests and the scheduler's timing transformations. Note velocity is not a separate model input; use a track or mixer gain for loudness.

An inline call works too: play (magenta "glass bells") "c4 e4 g4". Optional constant settings can be named:

keys = magenta "soft electric piano" { variation = 1.1, drums = false }
play keys "c4 eb4 g4 bb4"

variation ranges from 0.5 to 1.8 (default 1.3). drums = false suppresses drums; true lets the model decide whether to add them. Descriptions must be non-empty and at most 1000 characters. Audio-reference prompts and live modulation of these settings are later extensions.

Open Magenta Keys in Library for a four-chord example with a level macro. Sounds → Generate → Add to song as instrument inserts a binding and note pattern from the current description, variation and drum setting. Normal editor Undo restores the previous source.

Studio prepares each installed model when the song loads. The song instrument card in Sounds → Generate reports loading, readiness and errors. Wait for Instrument ready, then Play. Preparation is silent; starting playback before it finishes can miss early notes. Stop, pause and a track leaving its scene end its active generator; subsequent playback prepares it again. Edits that retain the track name and instrument recipe preserve model history. Closing the Sounds panel only stops its separate exploration session; the song's instruments keep playing. Multiple active generators share GPU time and may exceed realtime capacity; their cards report performance problems.

Model controls run in 40 ms frames, with additional buffering/inference delay. Studio requests each frame with the latest chord, keeping at most about 120 ms of audio ahead during playback. This reduces the delay added by Volta, but Magenta's own response still takes time: Google's latency overview reports roughly 200 ms for control response in its implementation. Total timing depends on the sound and machine; 40 ms frames do not mean 40 ms audible response. For tight rhythmic placement, Freeze a good phrase and trim/align the sample. Freeze preserves the performance you heard; it does not correct chord timing. For a live pattern, nudge can manually send notes earlier. At 120 bpm with the default four-beat cycle, this advances later changes by 250 ms:

t = play keys "[c4,eb4,g4] [d4,f4,a4]" |> nudge (-0.125)

Adjust by ear: the value is a fraction of a cycle, so its duration changes with tempo. The first attack cannot move before playback begins. Automatic timing calibration and scheduling ahead are not currently provided. Very fast gestures can merge, and exact pitch, attack timing and live replay are not guaranteed. Per-voice DSP graphs, envelopes, playOn, shred voice triggers and bounceOf cannot wrap a live Magenta source; the compiler explains how to use a track or frozen sample instead.

Freeze a song instrument#

In Sounds → Generate, open the song instrument's card, choose the last 1–30 seconds and click Freeze to sample. This saves audio actually heard, including its applied note/chord controls in the provenance sidecar. It saves the raw instrument before track effects, gain and master processing. Saving alone does not edit your song or stop the model.

Hear frozen sample auditions the complete clip. Use sample inserts its ordinary !"magenta-…" reference at the caret; Copy reference copies it. For example, a frozen phrase can play once in a portable song:

sound !"magenta-YOUR-SAVED-NAME" |> once

Replace the live track with a sample reference when preparing an offline export or browser song. Those hosts explicitly refuse live generators instead of rendering a silent substitute. A frozen WAV plays identically each time and can use the usual sampler, slicing and granular tools; include it with shared songs.

Get started#

1. Open Sounds → Generate.
2. On first use, choose Set up Magenta. Setup downloads the small model and its runtime, about 2 GB altogether. It needs an internet connection and the uv installer. The panel reports progress and offers cancellation. Subsequent generation is local; your descriptions are not sent to a service.
3. Describe a sound or choose a preset, then Start exploring. The first sound follows model loading and GPU warmup. Volta's selected output device plays the result while the current song is ducked.
4. Change the description while it plays. Variation adjusts how freely the model explores; Listen level changes monitoring volume. Allow drums removes the drum-suppression control.
5. Open Blend with another sound, describe a second sound and move Blend. Play a note · fine controls offers sustained pitch guidance, octave and description strength. These are model suggestions; exact timing, tuning and loop alignment are not guaranteed.
6. Choose the last 1–30 seconds and click Keep sample. The button becomes available after that much audio has been heard. The saved waveform appears below; Hear sample plays the whole clip and Stop sample stops it. Use sample inserts its sample-pattern reference at the editor caret. Keeping alone does not change the song, and insertion uses ordinary editor Undo.

Stop ends generation and releases the model. Closing/leaving Generate, changing songs, changing audio devices and the transport Stop also end the session. Its recent audio remains available for capture until a new session starts. Saved samples remain in Sounds → Samples and on disk.

Saved audio and portability#

Captures are 48 kHz stereo PCM16 WAV files in ~/Documents/Volta/samples, named magenta-<content digest>.wav. Capturing ends at Volta's consumed-audio cursor; it excludes the generator's queued future frames. Listen level does not change saved PCM. The audition applies short edge fades, while the WAV keeps the captured samples.

A sibling .magenta.json records Google's model revision and hashes, runtime identity, capture range, the controls actually applied and the WAV hash. Two different captures of the same duration receive different names. The source reference is ordinary Volta syntax, for example !"magenta-…"; no model is required to play a kept WAV. Include its audio when moving/sharing a project; the model itself does not run in the browser.

Runtime and troubleshooting#

The small mrt2_small model runs in an isolated Python process using Google's official Magenta 2.0.3 exported MLX wrapper. The environment pins MLX 0.31.1, Python 3.12 and all Python packages. Volta's audio callback only reads a bounded PCM ring; loading, inference, text encoding, resampling and file writes run elsewhere. The device-rate ring has 240 ms capacity and primes 120 ms before playback. The host requests one 40 ms frame at a time: it refills when queued audio falls to 80 ms, keeping at most about 120 ms ahead during playback. Silent preparation may queue 160 ms to cross the prime at every supported sample rate. The process pipe and packet queue cannot accumulate additional generated frames. A bounded 32-second history supports 30-second captures.

Runtime files live in ~/Documents/Volta/models/magenta-rt2. Errors point to magenta.log there. Check setup / repair rebuilds the isolated environment and rechecks/downloads model files without deleting saved samples. If uv is missing, install it using the link above and retry. A failed setup or missing model never substitutes synthetic test audio. A stalled worker times out; Stop remains available during loading and setup.

After changing an audio device's sample rate, audition a previously kept clip from Sounds → Samples. The Generate panel refuses a stale preview rendered at a different device rate.

Magenta's source is Apache 2.0. Google's model card supplies the CC BY 4.0 weight license and additional usage terms. Setup retains the model card, and capture sidecars credit Google. Stable Audio, TangoFlux, the larger Magenta model, audio-reference conditioning and physical MIDI input are not providers in this panel.