How to get high scan rates for stream in with non-looping stream out in T7 with Python

7 posts / 0 new

Tue, 10/06/2020 - 07:57

ilon

Hi,
I’m using LabJack T7:
Hardware Version: 1.35
Firmware Version: 1.0287
Bootloader Version: 0.96
Recovery Firmware Version: 0.6606
I would like to be able to stream in (at least 7 channels) and out simultaneously at high scan rates (at least 5000 – 10000 per channel). Also, I would like to stream out specific signals read from a file.
I’m using Python to write an application to do that. Thanks to the examples attached in Python_LJM_2019_04_03 library, I was able to stream in multiple channels at high scan rates. I was also able to load a signal from a file to an out buffer and stream it out, by modifying in_stream_with_non_looping_out_stream.py example. However, when I update the buffer out in the way, as in the example, I am only able to stream with lower scan rates. For streaming in 3 analog channels (with one stream out channel= FIO_STATE), I can’t get more than 2000 scan rate. When I want to add more stream in channels or core timer to my stream in, the possible, stable scan rate gets even lower and the LJM buffer overflows very fast (the device buffer is being cleared nicely). I was hoping that you could either take a look at my code and tell me what might be preventing me from getting higher scan rates, or suggest another way to update the buffer to stream out the data read from the file… That is, if that is why I am not able to stream at higher scan rates.

Description of my code:
My application is run from forum_test_t7 file. It allows to select in and out channels, scan rate and size of the out buffer. The stream will stop after clicking Ctrl+c. The StreamT7 class is responsible for continuously calling ljm.eStreamRead(handle) to read the stream, and logging. device_t7 file, is a modified ljm_stream_util.py example, where I changed create_out_context function to my own create_out_context_from_file, and added my configure_stream function to configure the stream in and out.
The testing version is hard coded to only stream in analog channels and use FIO0 and FIO3 as stream out channels.
Attachments:
my_waveform.txt – file with digital signal to stream out (to run it with the app, convert to csv)
forum_test_t7.py – test file that starts the stream (change INPUTS, SCAN_FREQUENCY,OUT_BUFFER_SIZE)
device_t7.py – file with functions that define how to update out buffer and how to configure stream
StreamT7 – class to create object that repeatedly calls ljm.eStreamRead

+example log files

File Attachment:

t7_logfile1000_with_core_timer.txt

t7_logfile5000.txt

t7_logfile3000.txt

t7_logfile2000.txt

Tue, 10/06/2020 - 15:14

LabJack Support

Stream-out will typically

Stream-out will typically only achieve the same maximum sampling rates as in-stream when you loop over a set number of samples; there is a typical communication overhead of around 0.6ms per modbus packet sent to the device from a host computer over USB and this can significantly limit your sample rate if you are dynamically sending updates for your waveform. During our testing, we have found similar limits of around 2000-3000Sa/s using the in_stream_with_non_looping_out_stream examples.

We recently created some functions in LJM (currently in beta) that help make stream-out easier and the AperiodicStreamOut functions work with faster stream rates. During testing, we observed stable stream-out waveforms at up to around 10,000Sa/s.

You can find the beta LJM install here:

https://labjack.com/support/software/installers/ljm

We have Python examples on our Github page that include these new functions (note that you will need to rebuild ljm-python from source since the Python wrapper was updated to include the new functions). I would recommend reading all of your data from file and bulk loading it with the WriteAperiodicStreamOut function before starting stream. The new functions have their own thread for doing work, so that should fix any issues with eStreamRead you are having due to streaming out:

https://github.com/labjack/labjack-ljm-python

Wed, 10/07/2020 - 00:52

ilon

Thank you for your fast

Thank you for your fast response. I will try the new library version to see if I can get faster stream rates with that.

Tue, 10/13/2020 - 04:36

ilon

Hi,

Hi,
I upgraded to beta LJM and I did some tests of aperiodic stream out. The speeds look great and very stable, and the buffer is not overflowing! However, I’m having hard times trying to understand how WriteAperiodicStreamOut works. In the example (stream_in_with_aperiodic_stream_out.py), the function is being called before I start the stream, and then cyclically, before every eStreamRead. It is regularly updating buffer with constant number (i.e. 512 samples, of the same values (increasing from 0 to 2.5)
1)   When I stream-in two channels at the same time at i.e. 5000 scan rate, I am getting 2500 samples per channel. What are the values for streamed-out signal above 512 sample? Is it last read value that is being replicated until the next read from the buffer? (cyclical_increase_of_voltage.png)
2)   The documentation (https://labjack.com/support/software/api/ljm/function-reference/stream-f...) mentions that buffer out max size is being allocated automatically. You recommended to load the WriteAperiodicStreamOut function with a file content before starting the stream. However, my files might often be longer than max number of samples the out buffer can hold, and I will have to update it. How can I do that? How often?
3)   Also, I don’t fully understand the sleep time calculation between every eStreamRead. It is calculated based on the user defined number of writes to the buffer. What if I want to stream for undefined amount of time (until user stops the stream)?
4)   Lastly, and unfortunately, my streamed-out signal does not look like the signal from the file at all…(streamed_out_sine.png) I tried to load WriteAperiodicStreamOut function with sequential data chunks from the file before every eStreamRead, but this does not seem to work. As a benchmark, I am streaming in a sine wave, to check if I am not losing any data. And that signal looks perfect. I would be really grateful if you could tell me what am I doing wrong. I attached my updated code.
I configure the stream within configure_stream function in device_t7_aperiodic.py, and call eStreamRead in read_stream function that is StreamT7_aperiodic class method. forum_test_t7_aperiodic.py is a test file
Attachments:
my_sine_wave.csv – file with signal to stream out
forum_test_t7_aperiodic.py – test file that starts the stream
device_t7_aperiodic.py – file with functions that define how to configure stream
StreamT7_aperiodic – class to create an object that repeatedly calls ljm.eStreamRead
+examples of streamed signal that was plotted afterwards (top channel is stream-in of sine wave, bottom channel is the signal read from a file and streamed-out using WriteAperiodicStreamOut)

File Attachment:

my_sine_wave.txt

forum_test_t7_aperiodic.py.txt

device_t7_aperiodic.py.txt

StreamT7_aperiodic.py.txt

cyclical_increase_of_voltage.PNG

streamed_out_sine.PNG

Wed, 10/14/2020 - 12:18

LabJack Support

Streaming in, you are

Streaming in, you are actually streaming each channel at the scan rate, the total sample rate seen by the device is numChannels*scanRate. The 512 samples written in the loop are arbitrary, it is just supposed to demonstrate that writing stream out data while doing other things can work. Using WriteAperiodicStreamOut writes samples to an internal software buffer; a worker thread monitors the device sample buffer, pops samples from the software buffer, and writes them to the device buffer such that the device buffer is ideally always full. If the software buffer runs out of samples to write you will end up draining the device buffer and when that happens the last DAC value you write will persist until more samples are written to the device buffer.
Are you referring to having more than 16384 bytes of data or more than 20 seconds worth of data? You do not need to worry about the device buffer. If your file has more than scanRate*20 samples, then you could write to an LJM configuration to increase the software buffer size or you could write samples while streaming in as shown in the example. The LJM configuration is LJM_STREAM_OUT_BUFFER_MAX_NUM_SECONDS which you can write using the function LJM_WriteLibraryConfigS:
https://labjack.com/support/software/api/ljm/function-reference/LJMWrite...
The sleep calculation is after the main program loop and not between eStreamRead calls. The sleep call is only there to warn users that you can write values using WriteAperiodicStreamOut and not have them output if you call eStreamStop before you can output all samples. writeOutTime = numOutSamples / scanRate. If you are letting the user stop the stream this sleep timer is not relevant for your application.
My guess for why your out stream is not working correctly is that eStreamRead is blocking for some time, around 500ms if you follow our API doc recommendation, which means you would only be feeding new data to stream out every 500ms. Fixes for that would be to bulk load your data to WriteAperiodicStreamOut before streaming, running a separate thread or using a callback to read stream data, decreasing the scansPerRead of eStreamRead to decrease the blocking time, or changing LJM configurations such that eStreamRead is not a blocking call. See these pages for more relevant information:
https://labjack.com/support/software/api/ljm/streaming-lots-of-9999-values
https://labjack.com/support/software/api/ljm/constants/ljmstreamscansreturn

Thu, 10/29/2020 - 08:57

ilon

Thank you for your detailed

Thank you for your detailed explanation. It really helped me better understand how WriteAperiodicStreamOut works. And I had no idea that it is possible to modify LJM stream out buffer size! So, thank you for telling me how to do that.
Now, with that knowledge, all I needed to do with my code, was:
-> increase LJM buffer out size as needed,
-> upload a whole file to WriteAperiodicStreamOut once (!) before I start the stream,
-> set a reasonable number of scansPerRead (I went with 1000),
-> and remove the unnecessary sleep time from my code.
That solved all my issues. The stream speeds are fantastic! I got 20,000Sa/sec while simultaneously streaming in 3 analog channels and streaming out digital FIO_STATE. Which I believe, is as good as speeds for stream in only.Without any buffer overflows or data loss. All beautiful and stable. Thank you for that simple and brilliant solution for stream out.

Wed, 01/13/2021 - 07:29

Ron.Gilliam

I am looking for someone to

I am looking for someone to assist with a project I have using T7 streaming several channels of data and generating operator interface to display various operator screens. If interested please contact [email protected]