Hello,
We are using a T7 and Matlab on a mac to to run an experiment. We read from two sensors and display the results on the screen online (continuously).
Software versions are:
macOS Sierra (v 10.12.6)
Matlab 2016b
Xcode 8.3.3 Build version 8E3004b
LJM Library version 1.17
We read the data from LJ using streamread. A minimal (hopefully) working code of what we do is below:
dev = LJ_Handle;
buffer = 2;
dev.streamRead;
dataAvailable=true;
while dataAvailable
if backlog<=buffer
dataAvailable=false;
else
[leverRead,backlog]=dev.streamRead;
end
end
%displayScene on the screen
Esentially, we use streamread in a while loop to get the first-in element of the stream (and do some non-computationally demanding downstream stuff with it). After the stream has been emptied, we refresh the scene that we want to show on the screen. This worked well for some time (in the first version of our experiment, ran a few months ago).
Now we plan to run a second version of the experiment. As far as we can tell, we are using the same software versions (there were no updates of the LJ library, no big macOS updates and no Xcode updates). However now things don't run as smoothly as they used to. Usually, the while loop can be completed within one refresh frame. But sometimes, it takes much longer (up to 120 msec) and too many frames are missed. This leads to a noticeable jump in our visual presentation. I attached a figure showing the difference (in msec) between frames using a code similar to the one I sent.
We played around with the settings that we could think of in the labjack that might help (resolution index, settling time and stream transfers per second). But none of them helped. I would like to avoid updating software versions etc because there are incompatibilities and we know that, at least in the past, our settings worked.
Ideally, however, there would be a way to dump the whole contents of the stream from the labjack to into matlab without having to use a while loop to read the data element-wise.
Any help would be welcome!
Thanks
Elisa
.png%3Fitok=PHGBxf6M "EEV Blog Reviews LabJack T7-Pro & Software Tools")
.jpg%3Fitok=B1YcuDFK)
.png%3Fitok=Ybt-OlgU "LabJack + JEDI One Can Create a Powerful Distributed DAQ Ecosystem")
Anecdotally, I've noticed my MacBook Pro oftentimes having more delays for longer periods. This seems mostly dependent on how many applications / Chrome tabs are open, but I've also assumed it's in part due to the security patches for Meltdown and Spectre. (They're timing-based attacks, so Apples security patches—which are software mitigation—prevents the clock from being read as accurately as it used to be able to and also reduce processor throughput by some significant percentage.)
In regards to actually improving your performance, I firstly recommend upgrading LJM due to these changelog items:
LJM 1.1801 - macOS: Improved USB stream start and stop speeds.
LJM 1.1803 - Increased theoretical USB/Ethernet stream speed.
You may need the 1.1900-development version, i.e. labjack_ljm_software_2018_11_02_beta.zip from 11/02/2018. (It was built with a older -mmacosx-version-min than the current Release version.)
Next, I recommend looking at this low-latency stream section:
http://labjack.com/support/software/api/ljm/function-reference/ljmestrea...
Some questions, if the above things don't fix the issue:
Which backlog are you using for "if backlog<=buffer"? DeviceScanBacklog or LJMScanBacklog?
If DeviceScanBacklog, that is more driver and T7 dependent. The extra slowdown could be related increased USB delay or more CPU usage.
If LJMScanBacklog, that is more user code dependent. Make sure extra delay wasn't added to your stream read loop routine. Delays between your stream read loops will make that buffer grow more. You can use LJM_GetHostTick() to measure the time between loop iterations.
For both backlog types, keep in mind that increasing the scan rate will create more of a backlog.
I'm not sure I understand. Do you want to read all stream data at once? If so, LJM_StreamBurst does that. (It doesn't return until stream is finished.)
Lastly, have you looked at the LJM Matlab examples? They're here:
https://labjack.com/support/software/examples/ljm/matlab
Thanks for your help!
First I tried to use the different backlog. As far as I understand, we were using the DeviceScanBacklog. Using instead the LJMScanBacklog didn't help because it is always 0. Might this reveal a potential problem?
I also tried updating the LJM Library, as you suggested. I tried with both the beta version and the stable release. In both cases I got a long error stack (I copied it below this message). So I downgraded back to version 1.17 in a little moment of panic.
After the upgrade-downgrade of the library, the behaviour changed a little bit. I attach again another figure with the new differences between frames refreshed. As far as I can tell, I had never seen delays of over 120 msec before, but now I do.
I am also using (as is described in the documentation you sent) ScansPerRead = 1 and LJM_STREAM_TRANSFERS_PER_SECOND = 999999999.
Finally, I had misunderstood the problem before. I thought that the lag happened because the while loop that read element-wise from the stream took too long. That was my mistake. I understand now that it's actually one single read that takes this long (which also explains why the lag happens in steps of either a couple of msec or 120 msec but nothing in between). Therefore, in principle I don't need the stream burst that you suggested, as there is no reason to think that reading the whole contents of the stream should take less time than reading out one single point. Sorry for my confusion.
Any other ideas of what I could test, try or do?
Many thanks
(Actually, I was wrong: we were using the LJMScanBacklog. It's DeviceScanBacklog that's always 0)
What's your scan rate and how many channels are you streaming?
What function call is seeing the delay?
A LJMScanBacklog of 0 is good. It means your loop is running fast enough. If LJMScanBacklog starts increasing, that means the device is sending bytes faster than your program is processing them.
I'm not familiar with building Matlab projects, but:
1. Can you do a full clean and rebuild of you Matlab project to make sure building works
2. Then, upgrade LJM as before, and do:
a. Try to do a full clean and rebuild
b. If that doesn't work, please try verifying that LJM 1.19 works by opening Kipling.
-We're using a scan rate of 1000 (And we check that this is true on starting the stream). We're streaming from two channels.
-The delay happens on the function streamRead (which in turn calls:
[err, data, DeviceScanBacklog , LJMScanBacklog]= calllib('libLabJackM','LJM_eStreamRead',obj.nr,obj.data,obj.DeviceScanBacklog,obj.LJMScanBacklog);
-(Maybe you my correction to my previous response came too late, it was actually DeviceScanBacklog that is always 0. LJMScanBacklog can get rather high).
-I can't build my project, because matlab is mostly an interpreted language. We aren't compiling anything.
Instead, is it possible that there is some incompatibility between the newest LJM and my old XCode ( 8.3.3 Build version 8E3004b)?
Thanks again
What connection type are you using? If Ethernet or WiFi, TCP will occasionally fail to send a packet, causing exponential backoff to occur for resending the packet. If that's what's happening, you could try to connect the device directly to your computer:
https://labjack.com/support/app-notes/networking/direct-connection-ether...
In any case, you could keep a buffer of stream data in your code. For example, it could aim to display data once every 15 ms (which I believe is faster than the human eye can see). If you collect 120 scans in your buffer, your displayed data can typically be 120 scans behind (and 120 ms behind). However, when there is a long delay, your program can still maintain the refresh rate by consuming data from the buffer.
In regards to LJMScanBacklog, if it's continually increasing as the program continues to run, that's bad. That means your code is not processing the stream data fast enough. You should probably increase the ScansPerRead parameter and process more data points at once to reduce overhead.
Can you try upgrading again and seeing if Kipling works?
In regards to your Matlab setup, I'm confused. In your output, I see (abbreviated):
Error using loadlibrary
Building libLabJackM_thunk_maci64 failed. Compiler output is:
/usr/bin/xcrun -sdk macosx10.12 clang ..... -o "libLabJackM_thunk_maci64.dylib" ...
libLabJackM_thunk_maci64.c:33:34: error: expected parameter declarator
EXPORT_EXTERN_C (__visibility__("default")))intLJM_Close(intHandleThunk(void fcn(),c...
So Matlab is trying to build something called libLabJackM_thunk_maci64.dylib. I also see that intLJM_Close (like other functions) does not have whitespace between int and LJM_Close. Maybe there was something strange going on with formatting when you copied and pasted, though.
If there's a bug in LJM that's causing this delay in eStreamRead, I'm not sure why you didn't see it before, but you'll need to be able to upgrade LJM.
Thanks for your help!
We are using a USB connection.
Thanks for the suggestion of buffering data. In our case that wouldn't work because the information on the screen corresponds to the position/rotation of our participant's arms, in (almost) real-time. If we kept a buffer of information, the movement displayed on the screen would be delayed relative to the actual movement and we would get into all sorts of unwanted problems.
Your suggestion of increasing the ScansPerRead helped, sort of. It reduced the time difference between two reads when we do a check with a minimal code (now the longest delays are of around 24 msec). However we still see lags/jumps in the display in our experiment. These happen much less often now, though. It's possible that the problem is not 100% fixed and if we sampled longer with our minimal test script, we would also get longer delays. But it could also be our fault, and we'll look some more into our code, perhaps we can solve it from our side.
However: I tried again to upgrade to the latest stable release of the LJM. I can't open Kipling (it gets stuck on the loading window, with a text that reads: "Extracting io_manager". Then I tried downgrading back to our old LJM and I still can't load it. I didn't try it immediately before upgrading, but Kipling used to work.
I tried to re-run the code with the new LJM version. I'm sorry for the cofusion: You're right. Matlab does seem to complile the LJ library when we call loadlibrary. But the scripts we wrote don't get compiled. So I can't rebuild my project. When I re-run our scripts I get, again, the same error I copied in my previous message. The missing spaces between int and LJM_close is also missing on matlab's console. I don't know if this is a problem in matlab's error report or there is a problem in the LJM itself. But it's not due to copy-pasting.
Sorry for the delay—our LJM expert is out sick for the time being. He'll probably get back to you on Monday.
For Kipling, try deleting the /usr/local/share/LabJack/K3 folder. You don't need to reinstall afterward. Does it start up correctly after that?
For getting an updated LJM version to run, let's try to get more debug information. Please try downloading the C/C++ examples:
https://labjack.com/support/software/examples/ljm/c
Since it you already have Xcode, you should be able to run the following commands without installing anything extra:
I would expect either make.sh or single_ain to fail. Please let me know which one fails and what the error message is.
I just ran a version of C_C++_LJM_2018-10-02/more/stream/stream_basic.c that runs at 1kHz with one scan per eStreamRead using LJM 1.1900 for 100 seconds. macOS 10.13.6. The longest time eStreamRead took, according to a LJM_GetHostTick placed immediately before and immediately after, was ~6 milliseconds. (I've attached that version.)
Also, I should mention that the Communication page of the T-Series Datasheet mentions that command-response mode is generally preferable for minimum-latency applications such as feedback control. Perhaps that's another option for you?
Edit: Added link.
Thanks and sorry for the delay.
I deleted /usr/local/share/LabJack/K3 folder. Now Kipling is still stuck on "Extracting static files" when opening.
I downloaded and ran the command you suggested. None of the two scrits seem to have failed. Below is the output I got in the terminal.
We don't want to use command-response solution, because we need to calculate the instantaneous angular velocity of people's arms (they rotate a capacitor, essentially). Because we simultaneously display on the screen what they are doing, we would only be able to sample the arm's angle ever ~16msec, with every screen refresh. That would not give us enough precision to measure speed (it would be enough to display it, of course, but we use the speed for other stuff and need a measurement that is as precise as possible).
I think I goofed when I told you to delete the /usr/local/share/LabJack/K3 folder—I should have said to delete everything in it (there may be a permissions problem). Please re-run the installer then try Kipling again.
If LJM 1.1900 programs are able to build and run, as your last comment shows, then it seems like there must be a problem in the Matlab layer.
Looking through your original error output, this seems suspicious:
Error parsing argument for function __attribute__ function may be invalid.
I'm wondering if Matlab (or something) does not know how to parse the __attribute__((__visibility__("default"))) attributes in LabJackM.h. Please try the following:
1. Edit /usr/local/include/LabJackM.h. Replace lines 42 - 46:
#define LJM_ERROR_RETURN __attribute__((__visibility__("default"))) int
#define LJM_LONG_LONG_RETURN __attribute__((__visibility__("default"))) long long
#define LJM_VOID_RETURN __attribute__((__visibility__("default"))) void
#define LJM_ERROR_STRING __attribute__((__visibility__("default"))) const char *
#define LJM_DOUBLE_RETURN __attribute__((__visibility__("default"))) double
With:
#define LJM_ERROR_RETURN int
#define LJM_LONG_LONG_RETURN long long
#define LJM_VOID_RETURN void
#define LJM_ERROR_STRING const char *
#define LJM_DOUBLE_RETURN double
2. Then make sure simple_ain can build and run successfully:
cd C_C++_LJM_2018-10-02/more/ain/
./make.sh -c # -c is for clean
./make.sh
./single_ain
This worked for me, but it would be good to check on your system.
3. Then try running your Matlab build again (after cleaning anything you can). The error output should at the very least be different from before.
If that works, it'll be a temporary solution and we can release a version of LJM that doesn't need such a workaround.
That solved all our problems :) Everything is ok now!
I re-ran the installer of the latest LJM library, now Kipling works again.
I made the changes you suggested in LabJackM.h and now Matlab can build the library and -as far as I could tell- there were no lags anymore. Short comment for future reference: for some reason our system (MacOS Sierra) didn't let me edit the contents of the file in a normal text editor, but it worked easily using vim.
Thank you, this was great help. You made us all very happy. We'll update the LJM library once you've included a solution that doesn't need the workaround.
That's good to hear! I'll update this thread when there is a version of LJM that doesn't need the workaround.
As of LJM 1.2000, LJM no longer needs the workaround since LabJackM.h no longer contains __attribute__ for macOS.
As of writing, it's available as a beta installer:
https://labjack.com/support/software/installers/ljm