DSP - Spectral Freeze

Specs

Overview

Algorithm

Spectral Freeze is based on the PaulStretch (http://hypermammut.sourceforge.net/paulstretch/) algorithm. I've taken the basic idea and added a number of real-time tweak-able parameters to enhance the playability of the algorithm. This give the algorithm a greater role in a live-performance context as you can shape the time modulation and spectrum modification in a few different ways depending on the context.

There are probably two primary mechanisms for freezing or stretching audio. The simplest is grain based: looping a small fragment of a time-based signal possibly with some envelope modification to smooth out the loop point transitions. This mechanism is probably more common in small digital effects boxes as it's a bit easier to implement in lower cost micro-controllers. The more complicated mechanism is based on the z-transform. 

This freeze algorithm uses the frequency-based z-transform. The z-transform (specifically, the Fast Fourier Transform) allows us to convert a discrete time-based signal (time domain) to a sequence of complex frequencies (frequency domain) and back again. By modifying the signal in the frequency domain before converting back into the time domain, we are able to achieve effects that would be impossible otherwise. 

Foundational Blocks

The PaulStretch algorithm breaks the time-based signal into individual blocks of a fixed number of samples. The FFT limits block sizes to powers of 2, so the sizes we have chosen as ideal are 8,192, 16,384, and 32,768. Each block is then converted to the frequency domain where the phases of each frequency are randomized (imagine 32,768 individual phasers) but the magnitudes are preserved. We then convert back into the time domain and that's our signal.

Well, not quite. The zero-time and N-time points of a z-transform are misrepresented since those points are not continuous. We could ignore this if we were simply taking continuous chunks of audio, one after another, FFT, then IFFT and back... but that's not much use... If we want to freeze or stretch our sound, we're going to be only moving forward our time signal by small amounts - or even none at all. Therefore, we need to make the result of each block to blend with the next block whether it's generated from audio that is one sample later or 16,384 samples later. 

We do this by taking a processed, time-domain block and multiplying it by another time-domain block whose values ramp up from 0.0 to 1.0 and back to 0.0 at the end of the block. This process is called windowing. The windowed block can be combined with any other windowed block as long as the overlap by N / 2 samples. If the windowing function (the function that defines the shape of the window) is just so, then there will be no perceivable amplitude modulation in the resulting audio. 

Then we have our signal - as far as PaulStretch is concerned. The nw2s freeze algorithm goes a bit further. 

Further Control

A stereo signal could easily be processed to have a single stereo output, but that seems a bit mundane. Instead, the nw2s freeze algorithm generates four channels from a singe mono audio input. These four channels are grouped as two pairs of left and right channels: the primary pair and secondary pair. While the primary and secondary outputs are generated from differing frequency magnitudes (more on that in a minute), the L and R output pairs are generated from the same frequency magnitudes, but with varying randomness in the phase angle. If both L and R were completely random (zero phase coherence), then the stereo image seems very distant and indistinct. If there is no difference in the phase from L to R, then there would be no stereo image. The width control adjust the phase coherence between L and R, spreading the image with larger values. 

Once you listen to the raw output of a signal that has had it's frequency-domain phases completely destroyed, it's easy to get a sense for how different types of signals are represented in the frequency domain. Dominant sustained tones tend to have large frequency domain magnitudes while transient sound energy has smaller magnitudes spread over a larger range of frequencies. The nw2s takes advantage of this characteristic to pull apart the more transient sounds from the sustained tones.

Using the mean threshold parameter, the user can gate any frequencies lower than a give level. When this parameter is set to 1.0, all frequencies lower than the average magnitude are muted. At 0.5, it mutes those less than half of the average. At 8.0, it mutes frequencies less than 8 times the average. 

Secondary Stereo Pair

So what do we do with all those frequencies that we are throwing away? This is where the second pair of outputs comes in handy... One parameter will cut the quieter frequencies which increases the sustained tones sent to the primary output (mean threshold down), while the other parameter (mean threshold up) will cut the louder frequencies which increases the transient and noisier sounds sent to the secondary pair of outputs.

Depending on the source material, extreme settings may produce too much attenuation and silence even the dominant and harmonics. To solve that issue, there is another set of parameters called "peaks-only" that breaks the frequencies into a set of buckets, finds the dominant frequency for that bucket and mutes all others. This can produce some very interesting effects on both the primary and secondary outputs since one tone per bucket will be represented (or disappeared) at all times. The peak width controls the bandwidth of the window from a few hertz to... a few more hertz. 

Although the mean threshold and the peak-only parameters are both available at the same time, they are in-effect mutually exclusive and should only be used one at a time. The following table lists the available buckets for peaks-only mode at a sample rate of 48kHz.

Global Parameters

Performance Modes

This algorithm can be used in a few different modes. Each mode has its own strengths, so use them as you see fit. Some of the performance modes have additional parameters specific to the way that mode works. 

The different performance modes are implemented as individual programs that can be built into a patch that's suited to that mode. Note that the freeze algorithm uses heavy DSP resources so may not work well with other DSP-heavy modules. 

Freeze

Real-time Freeze

Using the "freeze" algorithm in real-time mode seems a bit of a contradiction, however the effect can give audio processed in real-time an ethereal quality. The effect is similar to some of the crazier reverb sounds out there, but since it is not a reverb algorithm per se, the parameters and tweakability allow for some slightly different effects. 

This mode does have a latency related to the window size which needs to be taken into account when using with rhythmic audio. Experiment combining this effect with time-based effects and using longer or shorter windows to alter the timbre of processed audio. Don't expect millisecond responsiveness from this algorithm. 

Loop Stretch

The loop mode is similar to the Freeze mode except that it works on files rather than live audio.