Bringing in and outputting audio streams from/to numpy arrays

[snuq]

I am working on a program that uses pyo as the main audio processing library, but i want to be able to have audio generation and modification from other libraries in the pyo chain - for instance, I would like to use FluidSynth as an audio generator, and at some point i hope to implement VST plugins (wrapped into a pyo object somehow).
I am a bit stuck on the best way to incorporate this data into pyo.

For input, im thinking passing the data into a table and using TableRead or similar could work, but not sure how to ensure its written while pyo is processing the buffer, or if this would be the most efficient way to do this.

For output, i think there is a way to get the audio streams from objects, but again, i dont know how to syncronize this with the buffer.

If anyone can offer some advice, thank you in advance

[alexdrymonitis]

To get the audio to a NumPy array you can use getBuffer(). I'm not sure if all classes share this, probably, since all classes inherit from the PyoObject class. What I've done in my book is this:

tab = DataTable(s.getBufferSize())
np_array = np.asarray(tab.getBuffer())

About synchronizing, since you convert audio streams to NumPy arrays and back, there will be at least one sample block latency and I don't think you can avoid this.
For example, to write data to a NewTable() that you can then read with TableRead(), you will need a whole sample block to write that data to NewTable().
I don't think though that one sample block latency is all that bad.

[snuq]

sounds like a good start! i dont think a buffer of latency is too big a deal, im mostly concerned about running into audio stuttering or drops.
im wondering how i would synchronize the buffer reads, ie, how to know when to call getBuffer()?
would a Seq(time=buffer_size/output_frequency) powering a TrigFunc be accurate enough, or is there a better way?
looks like Server.setCallback() might do the trick, but it really would be better organizationally if this could somehow be called from the pyo wrapper object

[alexdrymonitis]

Seq() is part of the sample accurate timing classes, so it should be accurate as its sequencing is done in C, not Python. The same applies to TrigFunc(). You can of course use Server.callback() but that would happen at every sample block. If you want to control when your function is called, you're better off with TrigFunc().
Concerning stuttering or drops, that depends on what you'll do with the NumPy array. NumPy is fast, so it should probably be able to do its processing in between samples, but I haven't checked, so this is only speculation.

[snuq]

ah, guess i misunderstood the server callback documentation, i was thinking 'computation block' would be a buffer.

ill have to do some experimentation, but thanks for giving me a starting point!

also, side note, numpy is definitely fast enough - my first attempt at an audio engine (before i found pyo) was written entirely by processing numpy arrays and i had all the basics running in real-time (sample playback, synth waveforms, mixing, delay, even filtering)

[snuq]

been trying to get this working with the getBuffer() function.
right now, im just testing with a basic output to numpy array, multiply array by 0.5, back to pyo.
the documentation on getBuffer (https://belangeo.github.io/pyo/api/classes/_core.html#pyo.PyoObjectBase) states that it returns an object that can be directly modified in memory, which sounds ideal, but im running into synchronization issues...
my current code is:

    def __init__(self, input, buffer_size=1024, channels=2, mul=1, add=0):
        PyoObject.__init__(self, mul, add)
        self._input = input
        self._data = DataTable(size=buffer_size, chnls=channels)
        self.buffer = numpy.asarray(self._data.getBuffer())
        self._inputter = TableRec(input, self._data)
        self._tablewatcher = TrigFunc(self._inputter['trig'], function=self.buffer_filled)
        #self._data_output = DataTable(size=buffer_size, chnls=channels)
        self._out = TableScan(self._data, mul=mul, add=add)
        self._base_objs = self._out.getBaseObjects()

    def buffer_filled(self, *_):
        self.buffer*= 0.5
        #self._data_output.replace(list(self.buffer))
        self._inputter.play()

it actually works on the array directly, but it only works on every other buffer (sound rapidly shifts back and forth between full volume and half volume), not sure whats going on there, but if i can modify the buffer at just the right point in time, i think that method should work very efficiently.

if i try a less direct approach by uncommenting out those lines of code and passing self._data_output to the TableScan, the signal looks about right in my scope, but sounds like its being played through a ringmod (i think its dropping a few samples each buffer?)
this method seems much less efficient (for a start, having to convert the numpy array to a python list sucks...), but my cpu time only went up by a fraction of a %, so its probably not too bad

[alexdrymonitis]

How are you calling the buffer_filled() method? Do you use s.setCallback() for that? You only shared these two class methods. If you share more code, it will be easier to debug together.
Also, which of the two methods is less efficient? You mention that you have to convert the NumPy array to a Python list. That's what's happening in the code you shared above, in the line you have commented out. That's what you're saying? If this line is commented out, then how is the NumPy array passed back to a Pyo object?

[snuq]

How are you calling the buffer_filled() method? Do you use s.setCallback() for that? You only shared these two class methods. If you share more code, it will be easier to debug together.

buffer_filled is the callback from the TrigFunc() object stored in ._tablewatcher. there isnt any other code in the class that is relevant to this (just some variable set functions).

Also, which of the two methods is less efficient? You mention that you have to convert the NumPy array to a Python list. That's what's happening in the code you shared above, in the line you have commented out. That's what you're saying?

yeah, im concerned about the performance overhead having to convert the data into 'python' space (as opposed to pyo/numpy operating on the data in 'c' space, so to speak).

If this line is commented out, then how is the NumPy array passed back to a Pyo object?

thats how getBuffer works:
self.buffer = numpy.asarray(self._data.getBuffer())
returns a reference to the buffer, which can be modified directly in memory.
when i do:
self.buffer*= 0.5
its modifying the data directly and it doesnt need to be passed back in. see the documentation on the getBuffer command: https://belangeo.github.io/pyo/api/classes/_core.html#pyo.PyoObjectBase

the problem is, this in-memory modification only seems to happen on every other buffer because i cant figure out how to sync it.

[snuq]

it DOES seem that Server.setCallback functions are being called once-per-buffer (i ran some time.time() tests and it is about equal to buffer_size/output_frequency). the problem is, this is called at the START of the processing stack, and if i modify the buffer, its modifying the last one thats already been played.
however, making a copy of the buffer and passing that into a new table does actually work! (just delays the output by buffer_length samples)
here's what i have thats working:

class Test(PyoObject):
    buffers = []
    audio_callbacks = []

    def __init__(self, input, buffer_size=1024, channels=2, mul=1, add=0, audio_callbacks=None):
        PyoObject.__init__(self, mul, add)
        self.buffers = []
        self.audio_callbacks = audio_callbacks if audio_callbacks is not None else []
        self._input = input
        self._data = DataTable(size=buffer_size, chnls=channels)
        for channel in range(channels):
            self.buffers.append(numpy.asarray(self._data.getBuffer(channel)))
        self._inputter = TableFill(input, self._data)
        self._data_output = DataTable(size=buffer_size, chnls=channels)
        self._out = TableScan(self._data_output, mul=mul, add=add)
        self._base_objs = self._out.getBaseObjects()

    def buffer_filled(self, *_):
        for buffer in self.buffers:
            buffer *= 0.5  #whatever audio processing is needed is done here
        self._data_output.copyData(self._data)

    def play(self, dur=0, delay=0):
        if self.buffer_filled not in self.audio_callbacks:
            self.audio_callbacks.append(self.buffer_filled)
        return PyoObject.play(self, dur, delay)

    def stop(self, wait=0):
        if self.buffer_filled in self.audio_callbacks:
            self.audio_callbacks.remove(self.buffer_filled)
        return PyoObject.stop(self, wait)

and the 'audio_callbacks' is just a list of functions stored elsewhere that a callback passed to Server.setCallback() that iterates through and calls each one.

I still cant get over thinking that there has to be a more efficient way to do this... but at least its working and seems to be reasonably quick

edit: just found a better way to copy the data over to the second table, seems to be more efficient, reduced my cpu usage by 0.5%, and makes code cleaner too

[alexdrymonitis]

Glad you got it working! Would you like to share this better way of copying the data? I'm just curious.

[snuq]

I edited the code with the better version already, the copyData function doesnt need and conversions, and the data itself should stay out of python space as long as only numpy-based operations are performed on it