Forget Suno... He Builds His Own AI INSTRUMENTS — With Dr. Hugo Flores

From a waveform to a playable AI instrument

A voice transformed into the texture of a counting machine is a different musical idea from asking a service to generate a finished song. In this conversation, Dr. Hugo Flores explains how he builds instruments around a chosen sound palette, recorded gestures and interfaces that can be performed. His examples move between machine learning, sound design, sampling and live composition. They also show why the interface and the material used to train a model matter as much as the model name.

Flores describes two-stage audio generation as a response to the scale of digital audio. At 44.1 kHz, one second contains 44,100 sample values per channel. A model asked to generate every sample directly would have to handle a very high-dimensional sequence. The systems he discusses first compress audio, then generate in that smaller representation.

An encoder-decoder learns the compression stage. The encoder maps audio samples to a lower-resolution sequence; the decoder maps that sequence back to sound. Flores compares the idea loosely to MP3 compression, but stresses that the representation is learned: the model determines which information to retain in order to reconstruct audio. He calls the compact sequence audio tokens or latents. In the conversation, he describes it as roughly one hundred times smaller than the raw audio representation.

The second stage predicts or generates tokens. Flores compares the mechanism at a high level to a language model: a control, often a text prompt in commercial systems, guides a model that produces a sequence. The decoder then turns those tokens into audio. For the research setup he presents, the models are trained and run on computers in his lab; ChatGPT or another external language model is not generating the audio. The comparison to language models is an analogy for sequence prediction, not a claim that the audio system is simply a chatbot.

VampNet: mask tokens and let the model fill the gaps

The interview’s central generative example is VampNet, a music-oriented model Flores describes as filling missing sections in a token sequence. Start with audio that already exists and encode it. Then mask selected tokens, leaving gaps in the sequence, and ask the model to predict what belongs there. The source is not a blank page: the surrounding audio gives the model context for its reconstruction.

Flores demonstrates this with a recording of “Billie Jean.” After compression, parts of the token sequence are removed. The masked version sounds fragmented; VampNet then fills the gaps with its best approximation, based on what it learned during training. This is closer to transforming or reconstructing existing material than to composing an entire song from nothing.

The result reflects the training set. Flores says the general pretrained model was trained on a broad range of music. Because the model predicts plausible continuations, it tends toward common patterns in that material. He characterizes the default sound as broadly pop-like or average. The point is not that every generation sounds identical, but that a model’s learned distribution shapes what it considers a likely continuation.

His creative workflow changes that tendency through fine-tuning. He starts with the pretrained model, which has learned general structures in audio, then supplies a particular sound palette so that the model reflects those sounds. He is not trying to make the output resemble other people’s music. For the pieces and installations he discusses, he chooses source material himself and fine-tunes the system on those collected sounds.

The counting machine: voice as shape, machine as texture

The counting-machine demonstration makes the distinction concrete. Flores found a recording of a counting machine whose sound he liked and used it as the material for a palette. He then provides his voice as the input gesture. The voice supplies a changing contour; the fine-tuned VampNet reconstructs that contour using the mechanical character of the counting machine.

It helps to separate the two roles. The input is not just a prompt saying “make a counting machine,” and the machine recording is not simply played back unchanged. The model has learned a texture from the selected material and uses that learned palette to fill masked parts of the voice-derived token sequence. The output therefore depends on both what is performed and what the model has been taught to hear as a possible sound.

Flores describes a sound palette as anything from about five minutes to four, or even eight, hours of recorded or collected sound. That range reflects his explorations rather than a minimum specification for every project. The amount and character of the data should fit the intended transformation. He also connects this practice to sampling: he knows which recordings and pieces he selected and considers acknowledging those sources important. A custom dataset is an artistic choice and carries questions of selection and provenance along with it.

Token Telephone turns feedback into a way of listening

Token Telephone, an installation made with Stefan Moore, takes the transformation one step further: the output is fed back into the model as the next input, repeatedly. Each pass changes the sound a little. Over many passes, those small changes accumulate, making it possible to hear the model’s preferred directions or “resonances.” The process reveals not just an endpoint but a trajectory from the initial sound toward patterns the model handles confidently.

Flores describes a teaser made with the general model trained on a broad music collection. After roughly twenty minutes, very different starting points—including speech and noise—often drift toward a soft, relatively static piano chord. The exact path can vary, but the recurring destination makes the model’s learned average audible. In other demonstrations, a consonant or syllable can become a short rhythmic gesture, such as a hi-hat-like roll.

The installation has used several fine-tuned models so visitors can hear their own voices change. An opera model moves speech toward operatic textures; a choir model and an orchestra model lead to different transformations. Flores says he curated about three hours of favorite choral music for the choir palette. He describes a recorded voice passing through repeated transformations until it takes on a choral sound. That comparison shows why separate palettes produce distinct outcomes even when the basic feedback setup is similar.

Neural Tape Loop: record first, then perform the transformation

Neural Tape Loop grows from a limitation of VampNet. Flores does not present VampNet as a streaming effect that continuously processes live input with negligible latency. Instead, a sound is recorded and then transformed. He compares the workflow to tape: first record material, then cut, reshape or manipulate it. The model can slice a recording and add material from its learned dataset, but the user works with recorded segments rather than treating it as a conventional live insert effect.

The performance interface has eight model interfaces, each functioning like a track, plus a mixer. Flores can build a scene by generating separate elements, bringing them together, changing their levels and crossfading between them. Some elements come from text prompts. The demonstrated examples include typing “keyboard typing,” asking for a lion roaring, and generating bubbles. Other material can come from vocal input or acoustic sound. He combines these sources rather than treating one prompt as a complete composition.

In the quadraphonic setup, panners move sounds around the listening space. That spatial movement, along with mixing and timing, gives the performer control over how generated fragments form a live soundscape. The name “tape loop” captures both the record-then-transform logic and the exploratory, sometimes unwieldy quality Flores associates with the process. It is an instrument built around handling generated material, not a promise of instant response from every model.

Max, Python and a separate GPU computer

The Neural Tape Loop interface runs in Max, but it is only the front end of the system Flores describes. The Max patch communicates with a Python script on his laptop. That script passes a request to another Python process running on a computer with a gaming-class GPU; that machine runs the model and returns generated audio. The interface is therefore coordinating multiple parts, rather than containing all model weights and processing by itself.

Flores says VampNet is small enough to run on a single GPU in a gaming PC, citing a computer with an RTX 3080 as an example. Even so, the setup has several steps and depends on the separate machine and network being available. Before a performance, he worries about whether the computer in his Chicago lab is running and connected. He mentions hosted GPU services as a possible way to make models available remotely, but does not announce a finished commercial service. For musicians, this distinction matters: a Max patch alone is not the complete instrument, and the demonstrated installation depends on the associated Python processes and GPU system.

RAVE and BRAVE: a closer route to real-time response

For someone starting with less computing power, Flores points to RAVE, a real-time variational autoencoder. In the two-stage picture, RAVE corresponds more closely to the first stage: compressing and reconstructing audio. Instead of asking it only to reproduce the input faithfully, users can manipulate that reconstruction process to transform timbre. Flores calls it more limited than VampNet, but sees room to explore and notes that it can run locally without a separate server. He describes a Max patch and an external that can be installed, making it a comparatively approachable way to experiment with machine-learning instruments.

He also discusses BRAVE, a newer, faster development associated in the conversation with Franco Caspe. Flores says the improved speed makes audio input feel much better, especially for beatboxing transformed into drums. In the demonstration, a voice is converted into drum sounds in real time. He describes its latency as near zero, but gives no numerical measurement, so this is his qualitative account of the interaction rather than a benchmark.

That responsiveness comes with a data tradeoff. A focused voice-to-drums model needs a carefully curated and robust collection of clean drum sounds; Flores contrasts this with simply collecting five minutes of favorite music. He mentions roughly two hours of carefully selected drum material as the sort of palette this task might require. The data has to match the transformation goal, and low latency does not eliminate the work of choosing and preparing source sounds.

Make an interaction that supports your music

The conversation briefly introduces Stacco, an instrument by Nicola Bátó, as an example of physical interaction with a machine-learning model. A board contains magnetic sensors; moving magnets over it changes the model’s latent representation and therefore the sound’s trajectory. Flores likes that physical forces and gestures act directly on an otherwise abstract representation. He describes this as a gestural way to play with a model, not simply entering text into a box.

His broader advice is to make music with these systems, ask whether the result feels authored, and take it to performances or other listeners. The choice between a recording-based transformation, a live voice-to-drum instrument, and a magnetic gesture controller is a choice about interaction and timing. Flores also mentions building guitar pedals with Daisy Seed and a dub-siren synthesizer, but those are other projects rather than detailed build instructions in this conversation. For this interview, the useful design lesson is to begin with the musical action you want to perform, then choose a model, sound palette and interface that make that action possible.