Slow response using LJM over Ethernet | LabJack
 

Slow response using LJM over Ethernet

24 posts / 0 new
Last post
PierceNichols
PierceNichols's picture
Slow response using LJM over Ethernet

I'm using a T7-Pro over Ethernet with the LJM library, called from within C++ code, and it's running EXTREMELY slowly.

Reading the registers for six analog inputs using LJM_eReadAddresses takes 14-18 ms to finish, measuring from just before I call the function to when it return.

Writing two TDAC analog output registers using LJM_eWriteAddresses takes 213-219 ms to finish, measured the same way.

Any suggestions on what I should check to see why I'm getting such slow responses? I need to finish both parts in < 20 ms for my system to work the way I want it to.

PierceNichols
PierceNichols's picture
FWIW, the host is a 64-bit

FWIW, the host is a 64-bit Debian machine. I haven't observed this issue with our T4, and I also didn't initally see it on the T7-Pros, but once it started, I can't figure out how to make it start working properly. Any suggestions?

LabJack Support
labjack support's picture
"Write Read Loop with Config

"Write Read Loop with Config Speed Test eNames.exe" is a reference Windows application you can do for timing tests.  I just ran with the default values and get an average ms per iteration of about 0.4 milliseconds.

http://files.labjack.com/utilities/

Use this to get some times for comparison.

Also attached here as it might not be uploaded to the file server yet.

PierceNichols
PierceNichols's picture
For reasons that are tedious,

For reasons that are tedious, I can't easily run a Windows application against the T7. Do you have a Linux version available?

PierceNichols
PierceNichols's picture
Just ran c-r_speed_test from

Just ran c-r_speed_test from the LJM examples folder... it gives me an average time per iteration (reading eight AIN) of 1.419 ms. So why might LJM_eReadAddresses() take ten times that long to read a smaller number of addresses, and LJM_eWriteAddresses() take more than 100x as long?

 

LabJack Support
labjack support's picture
Setting TDACs does add an

Setting TDACs does add an additional small overhead to command-response times, but 200+ ms is unusually slow for setting both TDAC outputs. Please provide the following so we can try to reproduce the issue:

1. What version of LJM are you using?

2. What firmware version is on your T7?

3. A snippet of your code that has the issue. In particular, the TDAC LJM_eWriteAddresses call taking 200+ ms, including the TDAC configuration and register addresses used.

PierceNichols
PierceNichols's picture
LJM version: 2018_05_15_x86

LJM version: 2018_05_15_x86_64
Labjack Firmware Version: 1.0255

I'm limited in my ability to post my code because it's proprietary to my employers. I also wrote it to be generic, so things like the code and the addresses are separated. I know the register addresses are correct because I do not get errors and I get the outputs (and read the inputs) I expect to get. Other than that, the only setup I do is opening a handle for the T7.

Out of curiosity, I modified my code to use repeated calls to LJM_eWriteAddress() rather than a single call to LJM_eWriteAddresses().. and as a result it now writes both TDAC outputs in 2-4 ms (with a couple of outliers up to 7 ms) and the AIN read sped up as well. Why would that change produce a two order of magnitude increase in speed?
 

LabJack Support
labjack support's picture
Can you provide the TDAC

Can you provide the TDAC register addresses used with your LJM_eWriteAddresses call instead of code? Are you timing only the LJM_eWriteAddresses call?

So far I have been unable to reproduce the slow LJM_eWriteAddresses speeds when setting two TDAC outputs. I used the same LJM and firmware version as you. I would expect the one LJM_eWriteAddresses call to be faster than the two LJM_eWriteAddress calls.

PierceNichols
PierceNichols's picture
I'm writing to addresses

I'm writing to addresses 30000, 30002, 30004, and 30006.

The problem has recurred even with writing one at a time (switched to a different box), but only the first TDAC write of a group takes 200+ ms. When this issue manifests, it also slows down my call to read the analog inputs by a factor of ~10 (i.e. 1-2 ms becomes ~16 ms).

-p

PierceNichols
PierceNichols's picture
It's almost like the T7 doing

It's almost like the T7 doing some kind of time-consuming internal reconfiguration that takes a long time every time I read or write an address or group of addresses.

PierceNichols
PierceNichols's picture
I'm particularly disturbed by

I'm particularly disturbed by the fact that this problem occasionally seems to go away temporarily. Can you give me some other things to try to fix this issue?

PierceNichols
PierceNichols's picture
I also get this problem with

I also get this problem with DAC channels as well as the TDAC channels. It's always the first write of a set that takes 200+ ms.

LabJack Support
labjack support's picture
1. Please use Kipling to try

1. Please use Kipling to try resetting your T7 to factory defaults. Use the Power-Up Defaults tab.

2. Can you check if the problem still occurs with a USB connection? If not, that would narrow down the possibilities.

3. Please enable LJM debug logging and send us a log that captures the problem. Instructions for enabling debug logging are here:

https://labjack.com/support/software/api/ljm/function-reference/debuggin...

PierceNichols
PierceNichols's picture
Already tried #1, no change.

Already tried #1, no change.

#2 is inconvenient, so I'd like to avoid it if possible.

For #3, I added the following code:

if (LJM_WriteLibraryConfigS("LJM_DEBUG_LOG_MODE", 2)) {
    cerr << "Unable to enable debugging mode" << endl;
}
if (LJM_WriteLibraryConfigS("LJM_DEBUG_LOG_LEVEL", 1)) {
    cerr << "Unable to set debugging level" << endl;
}

The log file output from ten or so seconds of running is:

[Sep 25 23:12:56 2018] INFO - LJM version: 1.1803
[Sep 25 23:12:56 2018] PACKET - handle   1:  24 byte send    0x 48 E1 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:56 2018] PACKET - handle   1:  24 byte recv    0x 48 E1 00 00 00 12 01 4C
                                                                                       BD F0 4D 1B BE 55 F3 95 BC 51 9B 12 BC 4B 7C 1E
[Sep 25 23:12:56 2018] PACKET - handle   1:  16 byte send    0x 48 E2 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:56 2018] PACKET - handle   1:   8 byte recv    0x 48 E2 00 00 00 02 01 4C
[Sep 25 23:12:56 2018] PACKET - handle   1:  16 byte send    0x 48 E3 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:56 2018] PACKET - handle   1:   8 byte recv    0x 48 E3 00 00 00 02 01 4C
[Sep 25 23:12:56 2018] PACKET - handle   1:  24 byte send    0x 48 E4 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:56 2018] PACKET - handle   1:  24 byte recv    0x 48 E4 00 00 00 12 01 4C
                                                                                       BE AB 1C 88 BE 95 C8 99 BC 4E 0C E5 BC 50 79 78
[Sep 25 23:12:56 2018] PACKET - handle   1:  16 byte send    0x 48 E5 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:56 2018] PACKET - handle   1:   8 byte recv    0x 48 E5 00 00 00 02 01 4C
[Sep 25 23:12:56 2018] PACKET - handle   1:  16 byte send    0x 48 E6 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:56 2018] PACKET - handle   1:   8 byte recv    0x 48 E6 00 00 00 02 01 4C
[Sep 25 23:12:56 2018] PACKET - handle   1:  24 byte send    0x 48 E7 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:56 2018] PACKET - handle   1:  24 byte recv    0x 48 E7 00 00 00 12 01 4C
                                                                                       BE 7F B1 C2 BE 9D 35 EB BC 59 95 CC BC 51 F8 28
[Sep 25 23:12:56 2018] PACKET - handle   1:  16 byte send    0x 48 E8 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:56 2018] PACKET - handle   1:   8 byte recv    0x 48 E8 00 00 00 02 01 4C
[Sep 25 23:12:56 2018] PACKET - handle   1:  16 byte send    0x 48 E9 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:56 2018] PACKET - handle   1:   8 byte recv    0x 48 E9 00 00 00 02 01 4C
[Sep 25 23:12:56 2018] PACKET - handle   1:  24 byte send    0x 48 EA 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:56 2018] PACKET - handle   1:  24 byte recv    0x 48 EA 00 00 00 12 01 4C
                                                                                       BE 85 40 E4 BE 93 A0 39 BC 53 81 30 BC 4B FD 68
[Sep 25 23:12:56 2018] PACKET - handle   1:  16 byte send    0x 48 EB 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:57 2018] PACKET - handle   1:   8 byte recv    0x 48 EB 00 00 00 02 01 4C
[Sep 25 23:12:57 2018] PACKET - handle   1:  16 byte send    0x 48 EC 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:57 2018] PACKET - handle   1:   8 byte recv    0x 48 EC 00 00 00 02 01 4C
[Sep 25 23:12:57 2018] PACKET - handle   1:  24 byte send    0x 48 ED 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:57 2018] PACKET - handle   1:  24 byte recv    0x 48 ED 00 00 00 12 01 4C
                                                                                       BE 8A 63 E5 BE 90 7B 93 BC 4F 19 CF BC 4A 21 A1
[Sep 25 23:12:57 2018] PACKET - handle   1:  16 byte send    0x 48 EE 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:57 2018] PACKET - handle   1:   8 byte recv    0x 48 EE 00 00 00 02 01 4C
[Sep 25 23:12:57 2018] PACKET - handle   1:  16 byte send    0x 48 EF 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:57 2018] PACKET - handle   1:   8 byte recv    0x 48 EF 00 00 00 02 01 4C
[Sep 25 23:12:57 2018] PACKET - handle   1:  24 byte send    0x 48 F0 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:57 2018] PACKET - handle   1:  24 byte recv    0x 48 F0 00 00 00 12 01 4C
                                                                                       BE B4 D7 13 BE A0 45 E1 BC 53 ED CA BC 4F E3 7F
[Sep 25 23:12:57 2018] PACKET - handle   1:  16 byte send    0x 48 F1 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:57 2018] PACKET - handle   1:   8 byte recv    0x 48 F1 00 00 00 02 01 4C
[Sep 25 23:12:57 2018] PACKET - handle   1:  16 byte send    0x 48 F2 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:57 2018] PACKET - handle   1:   8 byte recv    0x 48 F2 00 00 00 02 01 4C
[Sep 25 23:12:57 2018] PACKET - handle   1:  24 byte send    0x 48 F3 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:57 2018] PACKET - handle   1:  24 byte recv    0x 48 F3 00 00 00 12 01 4C
                                                                                       BE 8C DE 0C BE BB 69 E6 BC 57 B4 DA BC 50 F0 6A
[Sep 25 23:12:57 2018] PACKET - handle   1:  16 byte send    0x 48 F4 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:57 2018] PACKET - handle   1:   8 byte recv    0x 48 F4 00 00 00 02 01 4C
[Sep 25 23:12:57 2018] PACKET - handle   1:  16 byte send    0x 48 F5 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:57 2018] PACKET - handle   1:   8 byte recv    0x 48 F5 00 00 00 02 01 4C
[Sep 25 23:12:57 2018] PACKET - handle   1:  24 byte send    0x 48 F6 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:57 2018] PACKET - handle   1:  24 byte recv    0x 48 F6 00 00 00 12 01 4C
                                                                                       BE 8C 54 08 BE 8F A9 CF BC 52 88 F5 BC 4C F5 A2
[Sep 25 23:12:57 2018] PACKET - handle   1:  16 byte send    0x 48 F7 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:57 2018] PACKET - handle   1:   8 byte recv    0x 48 F7 00 00 00 02 01 4C
[Sep 25 23:12:57 2018] PACKET - handle   1:  16 byte send    0x 48 F8 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:57 2018] PACKET - handle   1:   8 byte recv    0x 48 F8 00 00 00 02 01 4C
[Sep 25 23:12:57 2018] PACKET - handle   1:  24 byte send    0x 48 F9 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:57 2018] PACKET - handle   1:  24 byte recv    0x 48 F9 00 00 00 12 01 4C
                                                                                       BE 96 BA B0 BE 8E 94 7D BC 55 90 AD BC 50 B7 87
[Sep 25 23:12:57 2018] PACKET - handle   1:  16 byte send    0x 48 FA 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:58 2018] PACKET - handle   1:   8 byte recv    0x 48 FA 00 00 00 02 01 4C
[Sep 25 23:12:58 2018] PACKET - handle   1:  16 byte send    0x 48 FB 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:58 2018] PACKET - handle   1:   8 byte recv    0x 48 FB 00 00 00 02 01 4C
[Sep 25 23:12:58 2018] PACKET - handle   1:  24 byte send    0x 48 FC 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:58 2018] PACKET - handle   1:  24 byte recv    0x 48 FC 00 00 00 12 01 4C
                                                                                       BE B5 F1 BA BE A4 B2 45 BC 53 76 D8 BC 4F BA 20
[Sep 25 23:12:58 2018] PACKET - handle   1:  16 byte send    0x 48 FD 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:58 2018] PACKET - handle   1:   8 byte recv    0x 48 FD 00 00 00 02 01 4C
[Sep 25 23:12:58 2018] PACKET - handle   1:  16 byte send    0x 48 FE 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:58 2018] PACKET - handle   1:   8 byte recv    0x 48 FE 00 00 00 02 01 4C
[Sep 25 23:12:58 2018] PACKET - handle   1:  24 byte send    0x 48 FF 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:58 2018] PACKET - handle   1:  24 byte recv    0x 48 FF 00 00 00 12 01 4C
                                                                                       BE 85 BC D8 BE B4 B6 6E BC 53 05 13 BC 4A EB 51
[Sep 25 23:12:58 2018] PACKET - handle   1:  16 byte send    0x 49 00 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:58 2018] PACKET - handle   1:   8 byte recv    0x 49 00 00 00 00 02 01 4C
[Sep 25 23:12:58 2018] PACKET - handle   1:  16 byte send    0x 49 01 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:58 2018] PACKET - handle   1:   8 byte recv    0x 49 01 00 00 00 02 01 4C
[Sep 25 23:12:58 2018] PACKET - handle   1:  24 byte send    0x 49 02 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:58 2018] PACKET - handle   1:  24 byte recv    0x 49 02 00 00 00 12 01 4C
                                                                                       BE 92 71 35 BE 8E 7F 51 BC 50 07 B2 BC 4C CC 43
[Sep 25 23:12:58 2018] PACKET - handle   1:  16 byte send    0x 49 03 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:58 2018] PACKET - handle   1:   8 byte recv    0x 49 03 00 00 00 02 01 4C
[Sep 25 23:12:58 2018] PACKET - handle   1:  16 byte send    0x 49 04 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:58 2018] PACKET - handle   1:   8 byte recv    0x 49 04 00 00 00 02 01 4C
[Sep 25 23:12:58 2018] PACKET - handle   1:  24 byte send    0x 49 05 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:58 2018] PACKET - handle   1:  24 byte recv    0x 49 05 00 00 00 12 01 4C
                                                                                       BE B5 D6 BD BE A4 05 AC BC 52 F0 63 BC 4F BA 20
[Sep 25 23:12:58 2018] PACKET - handle   1:  16 byte send    0x 49 06 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:59 2018] PACKET - handle   1:   8 byte recv    0x 49 06 00 00 00 02 01 4C
[Sep 25 23:12:59 2018] PACKET - handle   1:  16 byte send    0x 49 07 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:59 2018] PACKET - handle   1:   8 byte recv    0x 49 07 00 00 00 02 01 4C
[Sep 25 23:12:59 2018] PACKET - handle   1:  24 byte send    0x 49 08 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:59 2018] PACKET - handle   1:  24 byte recv    0x 49 08 00 00 00 12 01 4C
                                                                                       BE 88 6E 1A BE B8 D0 65 BC 59 1E DB BC 51 48 54
[Sep 25 23:12:59 2018] PACKET - handle   1:  16 byte send    0x 49 09 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:59 2018] PACKET - handle   1:   8 byte recv    0x 49 09 00 00 00 02 01 4C
[Sep 25 23:12:59 2018] PACKET - handle   1:  16 byte send    0x 49 0A 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:59 2018] PACKET - handle   1:   8 byte recv    0x 49 0A 00 00 00 02 01 4C
[Sep 25 23:12:59 2018] PACKET - handle   1:  24 byte send    0x 49 0B 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:59 2018] PACKET - handle   1:  24 byte recv    0x 49 0B 00 00 00 12 01 4C
                                                                                       BE 90 54 4E BE 8E C7 8F BC 53 AF BB BC 4D 4D 8D
[Sep 25 23:12:59 2018] PACKET - handle   1:  16 byte send    0x 49 0C 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:59 2018] PACKET - handle   1:   8 byte recv    0x 49 0C 00 00 00 02 01 4C
[Sep 25 23:12:59 2018] PACKET - handle   1:  16 byte send    0x 49 0D 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:59 2018] PACKET - handle   1:   8 byte recv    0x 49 0D 00 00 00 02 01 4C
[Sep 25 23:12:59 2018] PACKET - handle   1:  24 byte send    0x 49 0E 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:59 2018] PACKET - handle   1:  24 byte recv    0x 49 0E 00 00 00 12 01 4C
                                                                                       BE 97 CD 95 BE 8E AC 3F BC 4F E3 7F BC 4A 26 CD
[Sep 25 23:12:59 2018] PACKET - handle   1:  16 byte send    0x 49 0F 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:12:59 2018] PACKET - handle   1:   8 byte recv    0x 49 0F 00 00 00 02 01 4C
[Sep 25 23:12:59 2018] PACKET - handle   1:  16 byte send    0x 49 10 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:12:59 2018] PACKET - handle   1:   8 byte recv    0x 49 10 00 00 00 02 01 4C
[Sep 25 23:12:59 2018] PACKET - handle   1:  24 byte send    0x 49 11 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:12:59 2018] PACKET - handle   1:  24 byte recv    0x 49 11 00 00 00 12 01 4C
                                                                                       BE B4 D6 1A BE A9 01 3E BC 4D 52 B9 BC 48 36 57
[Sep 25 23:12:59 2018] PACKET - handle   1:  16 byte send    0x 49 12 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:13:00 2018] PACKET - handle   1:   8 byte recv    0x 49 12 00 00 00 02 01 4C
[Sep 25 23:13:00 2018] PACKET - handle   1:  16 byte send    0x 49 13 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:13:00 2018] PACKET - handle   1:   8 byte recv    0x 49 13 00 00 00 02 01 4C
[Sep 25 23:13:00 2018] PACKET - handle   1:  24 byte send    0x 49 14 00 00 00 12 01 4C
                                                                                       00 00 02 02
                                                                                       00 00 00 02
                                                                                       00 00 06 02
                                                                                       00 00 04 02
[Sep 25 23:13:00 2018] PACKET - handle   1:  24 byte recv    0x 49 14 00 00 00 12 01 4C
                                                                                       BE 84 70 41 BE B2 12 E8 BC 57 FD 40 BC 50 5A 71
[Sep 25 23:13:00 2018] PACKET - handle   1:  16 byte send    0x 49 15 00 00 00 0A 01 4C
                                                                                       01 75 30 02 00 00 00 00
[Sep 25 23:13:00 2018] PACKET - handle   1:   8 byte recv    0x 49 15 00 00 00 02 01 4C
[Sep 25 23:13:00 2018] PACKET - handle   1:  16 byte send    0x 49 16 00 00 00 0A 01 4C
                                                                                       01 75 32 02 00 00 00 00
[Sep 25 23:13:00 2018] PACKET - handle   1:   8 byte recv    0x 49 16 00 00 00 02 01 4C
 

LabJack Support
labjack support's picture
I don't see anything in the

I don't see anything in the log that gives a hint what's happening, but I also don't see TDAC registers being written in groups—I only see TDAC0 and TDAC1 being written one at a time.

1. Please try disconnecting everything from the T7 except the USB power and Ethernet cable. Does the problem still occur (with writing to DACs)?

2. Does the problem still occur if you write to TEST_FLOAT32 in a group (i.e. write to TEST_FLOAT32 / address 55124 four times at once with eWriteAddresses)?

3. Please try running the attached program. Does it replicate the problem? By default it writes to LJM_FLOAT32 four times at once. If that doesn't replicate the problem, does switching to writing to TDACs or DACs cause the problem? Does it take more iterations for the problem to occur? In general, how do you need to modify the attached program to replicate the problem (if you need to at all)?

When I ran the program (you can copy/paste it into c-r_speed_test.c in the examples download to help compile it), I found that the longestIteration was generally between 14 and 24 ms. I tried it with 2 frames writing to DAC0 and DAC1 and also tried it with 2 frames writing to TDAC0 and TDAC1, but I never saw the longestIteration be greater than 34 ms over some 20 runs. Switching to writing one frame didn't make much of a difference to the longestIteration measurement. (Being pretty noisy, this makes sense.) With a longestIteration of around 20 ms, the average ms per iteration was still low—e.g. around .5 to .8 ms for 4 frames writing to TEST_FLOAT32. Similar for 2 frames writing to DACs.

4. Since Ethernet latency is unpredictable, it's possible that there is a network issue. You could try to reduce network complexity by connecting your computer and the T7 to a separate router, or you could try connecting your T7 directly via Ethernet to your computer.

File Attachment: 
PierceNichols
PierceNichols's picture
The log out above was with

The log out above was with individual writes -- the problem occurs, at least some of the time, with both single and group writes. If I do several single writes ina a row, it happens only with the first write. Frustratingly, the issue comes and goes without apparent rhyme or reason. I've tried shutting things down when it was happen, with no change in the delay. I've also seen it run correctly with everything in our larger system up and running. I've observed it with two different units. It doesn't seem to be happening today, which means I'm not in a position to tell you if the attached program does anything

The delay, when present, is constant within a range of only ~10ms. If it happens, it's always on the first write in a series of writes. I've already tried removing everything else from the network, and as much as possible , and nothing else on the network appears to be affected by any similar delay.

Beyond the write problems, I'm also getting a longish delay on analog reads -- ~15 ms to do a group read of half a dozen channels.

PierceNichols
PierceNichols's picture
Today, I was able to

Today, I was able to reproduce the 'slow' mode again and run the provided speed test code. It returned an average iteration of ~2.5 ms with a longest iteration of ~90 ms. Interestingly, immediately after running it, my interface code runs at full speed and doesn't appear to have re-entered 'slow' mode again. Thoughts?

LabJack Support
labjack support's picture
Initial delays

Initial delays

Are you using an IP address as the Identifier to LJM_Open/LM_OpenS?

During the first LJM_Open/LM_OpenS, LJM asynchronously loads a constants file from disk. This takes around 75 ms for me. Usually this is good because the constants file will be loaded by the time LJM returns a device handle, but if an IP address is used as the Identifier, LJM can return the device handle immediately before the constants loading can complete. Since read/write operations require the constants to be loaded, a delay can occur while the constants are still being loaded. To work around this, your code can perform an operation to make sure the constants file is loaded. This could be any read or write to the device, or you could call LJM_NameToAddress any time before the first read or write that has critical timing:

    int addr, type;
    LJM_NameToAddress("AIN0", &addr, &type);

Intermittent delays

How often are you seeing delays of ~15 ms? Device I/O is subject to intermittent failure. No packet is guaranteed to be sent and received successfully, but device I/O—especially with regards to networks—still works because the chances of most packets sending successfully is good. TCP (which T7s use) is responsible for retrying failed packets with exponential backoff between retries. ~15 ms for a round trip time seems reasonable for a retry when either the command or response fails the first time.

I did a quick test reading 6 AINs 10k times in a row:

Ethernet via the network
    longestIteration: 111 ms
    Approximate average time per iteration: 8.455200 ms
    List of times 15 ms per iteration or longer (24 iterations were 15 ms or longer):
        54
        47
        19
        15
        15
        18
        16
        15
        15
        16
        22
        19
        16
        17
        95
        33
        19
        111
        15
        17
        15
        21
        15
        15

Ethernet via the network
    longestIteration: 85 ms
    Approximate average time per iteration: 7.708700 ms
    List of times 11 ms per iteration or longer:
        60
        46
        17
        26
        85
        11

USB
    longestIteration: 74 ms
    Approximate average time per iteration: 7.596100 ms
    List of times 10 ms per iteration or longer:
        62
        49
        74
        26

This test was done on a macOS laptop with many applications running.

PierceNichols
PierceNichols's picture
I am using the IP address as

I am using the IP address as an identifier, but ff it was just the first operation after making a connection, it's unlikely I'd even notice that there was a problem. My code reads a group of AIN channels and then immediately thereafter writes a series of TDAC channels. When it's in 'slow' mode, *EVERY* time it switches from reading to writing, it takes ~220 ms to make the first write of the group. When it is in 'slow' mode, the timing is very consistent -- there are no frames where the first write takes less than 210 ms or more than 230 ms. I have not been able to detect any pattern to when the T7-Pro enters 'slow' mode versus expected performance.

Is it possible that the T7 is disconnecting and forcing a reconnect every time I do a group read of AIN registers?

 

LabJack Support
labjack support's picture
I still haven't been able to

I still haven't been able to reproduce the "DAC 210 to 230 ms" problem.

1. What are the LEDs doing when the delay happens?

  • Is STATUS doing anything? It should be a constant green.
  • Is the orange Ethernet LED stopping blinking during those ~210 to 230 ms?

2. Please disable WiFi using Kipling. Please also make sure you're not running any Lua scripts on startup using Kipling.

3. Are you dynamically loading to LJM using dlopen?

4. I've modified the c-r_speed_test.c example to switch between reading AINs and writing DACs. It pauses execution if any read or write operation took longer than 200 ms.

You'll need to modify the IP address. You can also uncomment Print_eNames on line 101 for more output.

4.a) Does it cause the "DAC 210 to 230 ms" problem to occur? If not, can you modify it so that it does cause the problem?

4.b) Once it does cause the problem, please try to read CORE_TIMER before and after your DAC writes. The commented out <blah>DAC_time variables and SpeedTest show how to do this. (You can just uncomment those lines if the program causes the problem without modification.) The objective here is to see how long the DAC writes take.

If the DAC writes ever take 210 to 230 ms:

  • The delay is on the device during writing to DACs.

If the DAC writes don't ever take 210 to 230 ms:

  • The delay could still be on the device on the Ethernet stack before Modbus packets are handled (or anytime before / after the CORE_TIMER reads). To test in this situation, it would probably be best to try USB.
  • The delay could be in the network. Seems unlikely since you've found that it consistently only happens when you change something that has nothing to do with the network (read vs write).
  • The delay could be in LJM. Seems unlikely since your computer's CPU is fast, but if there's a strange synchronization bug in LJM, it's possible. You could try the LJM_WriteRaw and LJM_ReadRaw functions to reduce the possibility of this. You can use the packets in the log above to determine what packets you should send / receive.

5. Does the "DAC 210 to 230 ms" problem occur with different firmware versions? Try installing a really old one.

 

For the AIN reads:

6. What range and resolution are you using? Which registers are you using to write your AIN configurations?

File Attachment: 
PierceNichols
PierceNichols's picture
1) The green light is solid

1) The green light is solid on, and there doesn't appear to be an interruption to the blinking of the orange light, although I'm not confident I would notice.

2) & 3) I confirmed both of these things as part of my own troubleshooting.

4) & 5) It's misbehaving today (it wasn't yesterday), so I will be able to give both of these a spin and I will report back.

6) Range and resolution are factory defaults -- I haven't touched them.

LabJack Support
labjack support's picture
My wording of #3 was bad, it

My wording of #3 was bad, it should've been:

3. Are you dynamically loading LJM in your application using dlopen? If you are, there's a chance you could be loading LJM multiple times, which could incur constants file loading delays. (My test for constants file loading was done on a fairly fast computer with a solid state drive, so my ~75 ms could take longer for you.)

6. I would expect a T7-Pro with default AIN configs to take 3.5 ms per sample, plus overhead for the communication/etc. Data rates are documented here:

https://labjack.com/support/datasheets/t-series/appendix-a-1

PierceNichols
PierceNichols's picture
On #3, I'm not doing any

On #3, I'm not doing any dynamic loading of libraries -- it appears to load the shared library at the start of execution. Loading is from an SSD, and the host machine I'm using is most likely faster and has more RAM then your test machine.

I tested #5 with a rollback to 1.0228 (attempts at earlier versions failed with hardware incompatibility, although I didn't try every option). There was no change in behavior.

On #6, 3.5 ms/sample would correspond to what I'm seeing in 'slow' mode. In normal mode, the read operation takes 0-2 ms (trying every 20 ms).

LabJack Support
labjack support's picture
Have you been able to test #4

Have you been able to test #4 yet?