We have found an interesting behavior with the T7. We have it setup to sample 13 channels in streaming mode and send 4 samples per channel (13*4) back. Almost like clockwork after 20-22 minutes for days on end we will stop getting data from the T7 and have to reconnect to reestablish the data feed. Is there something inside the unit that would be causing this or some configuration we are missing or not setting that can ensure that the connection can stay stable for longer than 20 minutes at a time?
Cheers,
.png%3Fitok=PHGBxf6M "EEV Blog Reviews LabJack T7-Pro & Software Tools")
.jpg%3Fitok=B1YcuDFK)
.png%3Fitok=Ybt-OlgU "LabJack + JEDI One Can Create a Powerful Distributed DAQ Ecosystem")
LabJack devices stream indefinitely unless there's a reason to stop. E.g. until an error occurs, until stream stop is called, or until the end of the burst scans is reached. If you're not calling stream stop (from your process or from another process) and you're not using burst mode, what is the error code?
Have you enabled the watchdog? It's not reset by normal spontaneous stream. See "When Using Stream":
https://labjack.com/support/datasheets/t-series/watchdog
In our case, no we don't use the watchdog timer in any way and we don't use burst streaming.
Out read loop is very similar to the configuration used in the stream_callback python sample. Same labjack initialization config except for the 13 channels and 4 samples per channel. In the call back we are using the eStreamRead method to pull the 52 doubles and it works like clock work for 20-22 minutes. We have a monitor on our side looking for any loss of data and we have a tolerance of 3 seconds before we will close and reconnect. every 20-22 minutes that callback stop receiving data via eStreamRead without error or exception.
What are the LEDs doing? The STATUS led blinks continuously while stream is running.
If you try to communicate with the device after the callback stops, what happens? Is the device reachable? If you're using Ethernet/Wifi, try using ping to ensure the device is reachable.
Does it still occur if you run one of our stream callback examples with the time set to ~30 minutes? I've run a 30 minute test with 13 channels with a slightly modified stream_callback.py, attached. It ran successfully for 1800080 ms. The altered lines are as follows:
...
handle = ljm.openS("ANY", "ETHERNET", "ANY")
...
si.aScanListNames = ["AIN0", "AIN1", "AIN2", "AIN3", "AIN4", "AIN5",
"AIN6", "AIN7", "AIN8", "AIN9", "AIN10", "AIN11", "AIN12"]
...
si.scansPerRead = int(13 * 4)
si.streamLengthMS = 1000 * 60 * 30
...
From what I can tell the LEDs blink as expected. We are always connected via Ethernet to the device but I will have to do some work to hack up a test to try what you are asking. By default we kick the connection over and reestablish it and the connection always seems to come back without issue... it just appears that the device stops sending data back periodically. I have attached a stripped down version of our implemenation if you wanted to see what we are doing. Main difference between the sample and what we are doing is the threading out and monitoring of the collection.
The following is the output we get from it and you can see it seems to freeze up for at least 3 seconds every 22 minutes consistently. As I said, once we timeout at 3 seconds we kill the connection and reconnect.
19:02:48: Timed out waiting for data
19:02:48: Closing labjack at 10.6.0.102...
19:02:48: Labjack at 10.6.0.102 closed
19:02:48: Opening labjack at 10.6.0.102...
19:02:48: Opened LabJack with device type: 7, connection type: 3, serial number: -, address: 10.6.0.102:502
19:24:42: Timed out waiting for data
19:24:42: Closing labjack at 10.6.0.102...
19:24:42: Labjack at 10.6.0.102 closed
19:24:42: Opening labjack at 10.6.0.102...
19:24:42: Opened LabJack with device type: 7, connection type: 3, serial number: -, address: 10.6.0.102:502
19:46:36: Timed out waiting for data
19:46:36: Closing labjack at 10.6.0.102...
19:46:36: Labjack at 10.6.0.102 closed
19:46:36: Opening labjack at 10.6.0.102...
19:46:36: Opened LabJack with device type: 7, connection type: 3, serial number: -, address: 10.6.0.102:502
20:08:29: Timed out waiting for data
20:08:29: Closing labjack at 10.6.0.102...
20:08:29: Labjack at 10.6.0.102 closed
20:08:29: Opening labjack at 10.6.0.102...
20:08:29: Opened LabJack with device type: 7, connection type: 3, serial number: -, address: 10.6.0.102:502
20:30:23: Timed out waiting for data
20:30:23: Closing labjack at 10.6.0.102...
20:30:23: Labjack at 10.6.0.102 closed
20:30:23: Opening labjack at 10.6.0.102...
20:30:23: Opened LabJack with device type: 7, connection type: 3, serial number: -, address: 10.6.0.102:502
20:52:17: Timed out waiting for data
20:52:17: Closing labjack at 10.6.0.102...
20:52:17: Labjack at 10.6.0.102 closed
20:52:17: Opening labjack at 10.6.0.102...
20:52:17: Opened LabJack with device type: 7, connection type: 3, serial number: -, address: 10.6.0.102:502
21:14:11: Timed out waiting for data
21:14:11: Closing labjack at 10.6.0.102...
21:14:11: Labjack at 10.6.0.102 closed
21:14:11: Opening labjack at 10.6.0.102...
21:14:11: Opened LabJack with device type: 7, connection type: 3, serial number: -, address: 10.6.0.102:502
21:36:04: Timed out waiting for data
21:36:04: Closing labjack at 10.6.0.102...
21:36:04: Labjack at 10.6.0.102 closed
21:36:04: Opening labjack at 10.6.0.102...
21:36:04: Opened LabJack with device type: 7, connection type: 3, serial number: -, address: 10.6.0.102:502
21:57:58: Timed out waiting for data
21:57:58: Closing labjack at 10.6.0.102...
21:57:58: Labjack at 10.6.0.102 closed
21:57:58: Opening labjack at 10.6.0.102...
21:57:58: Opened LabJack with device type: 7, connection type: 3, serial number: -, address: 10.6.0.102:502
Something like power loss or a packet being sent to the device to stop stream could cause the device to stop streaming. That would cause the LEDs to stop blinking.
My previously attached stream_callback.py test demonstrates that the LabJack can stream for over 22 minutes. If you run it, does it successfully stream the entire time? There may be something with your network, with the power supply, etc.
Please try running that in conjunction with a continuous ping test. Does ping get successful responses from 10.6.0.102 for all 30 minutes?
I can certainly try a few things on my end. Does the device have any sort of advanced logging capabilities device side that might have some insight?
No, there is no event or debug logging on the device.
LJM does have (host-side) debug logging if you'd like to try that:
https://labjack.com/support/software/api/ljm/function-reference/debugging-functions
I would be happy to take a look at that debug log, if you attach one here (or email it in to [email protected]).
hmmm.. I have tried debug logging but I can't seem to get/set the log file name. Even trying to read constants.DEBUG_LOG_FILE throws and exception. Other DEBUG constants seem to work fine. Is there some other constant I should be using?
>>> import labjack.ljm
>>> labjack.ljm.readLibraryConfigS(labjack.ljm.constants.DEBUG_LOG_FILE)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/mcarle/source/platform/env/lib/python3.7/site-packages/labjack/ljm/ljm.py", line 2522, in readLibraryConfigS
raise LJMError(error)
labjack.ljm.ljm.LJMError: LJM library error code 1299 LJME_INVALID_CONFIG_NAME
>>> labjack.ljm.readLibraryConfigS(labjack.ljm.constants.DEBUG_LOG_MODE)
1.0
>>> labjack.ljm.readLibraryConfigS(labjack.ljm.constants.DEBUG_LOG_FILE_MAX_SIZE)
50000.0
I think you can ignore that last message from me... not sure what is going on there but managed to work with the default log
I have attached the entire log file but this is what seems to be happening when the connection needs to be reset... Not sure what that transaction ID is but it seems to be rolling over...
03 00 03 00 03 00 03 00 03 00 03 00 03 00 03 00 03 00 03 00 03 00 04 00 01 00 03 00 03 00 03 00 03 00 03 00 03 00 03 00 04 00 04 00 04 00 03 00 03 00 01 00 03 00 03 00 03 00 03 00 04 00 03 00 04 00 03 00 02 00 03 00 03 00 02 00 01
[Jul 20 20:16:22 2020] STREAM_PACKET - handle 2: 120 byte recv 0x FF FD 00 01 00 72 00 4C
10 00 00 00 00 00 00 00 00 03 00 03 00 04 00 03 00 03 00 04 00 03 00 03 00 05 00 04 00 03 00 03 00 01 00 03 00 02 00 03 00 03 00 03 00 02 00 03 00 02 00 03 00 02 00 03 00 04 00 01 00 03 00 01 00 03 00 03 00 03 00 04 00 04 00 03 00 03 00 02 00 03 00 04 00 01 00 03 00 02 00 03 00 03 00 03 00 03 00 04 00 03 00 03 00 03 00 03 00 03 00 01
[Jul 20 20:16:22 2020] STREAM_PACKET - handle 2: 120 byte recv 0x FF FE 00 01 00 72 00 4C
10 00 00 00 00 00 00 00 00 03 00 03 00 05 00 03 00 03 00 04 00 03 00 03 00 02 00 03 00 02 00 03 00 01 00 03 00 02 00 02 00 03 00 03 00 03 00 03 00 03 00 03 00 03 00 03 00 03 00 01 00 04 00 02 00 03 00 03 00 04 00 03 00 03 00 03 00 03 00 02 00 04 00 03 00 01 00 02 00 03 00 03 00 03 00 03 00 03 00 03 00 03 00 03 00 03 00 04 00 03 00 01
[Jul 20 20:16:22 2020] STREAM_PACKET - handle 2: 120 byte recv 0x FF FF 00 01 00 72 00 4C
10 00 00 00 00 00 00 00 00 03 00 03 00 04 00 04 00 03 00 03 00 03 00 03 00 03 00 03 00 03 00 03 00 01 00 04 00 03 00 04 00 03 00 01 00 02 00 03 00 03 00 03 00 03 00 03 00 01 00 01 00 02 00 03 00 03 00 03 00 03 00 03 00 02 00 03 00 03 00 04 00 04 00 03 00 01 00 04 00 04 00 03 00 04 00 03 00 04 00 03 00 03 00 03 00 02 00 04 00 04 00 01
[Jul 20 20:16:22 2020] ERROR - TransactionID was 65535, but was expected to be 0
[Jul 20 20:16:22 2020] TRACE - Disabling device stream
[Jul 20 20:16:22 2020] PACKET - handle 2: 16 byte send 0x A1 E5 00 00 00 0A 01 4C
01 13 7E 02 00 00 00 00
[Jul 20 20:16:22 2020] PACKET - handle 2: 8 byte recv 0x A1 E5 00 00 00 02 01 4C
[Jul 20 20:16:25 2020] TRACE - Disabling device stream
[Jul 20 20:16:25 2020] PACKET - handle 2: 16 byte send 0x A1 E6 00 00 00 0A 01 4C
01 13 7E 02 00 00 00 00
[Jul 20 20:16:25 2020] PACKET - handle 2: 9 byte recv 0x A1 E6 00 00 00 03 01 CC
03
[Jul 20 20:16:25 2020] PACKET - handle 2: 12 byte send 0x 00 E7 00 00 00 06 01 4C
00 D6 D8 04
[Jul 20 20:16:25 2020] PACKET - handle 2: 16 byte recv 0x 00 E7 00 00 00 0A 01 4C
0A 3C 00 03 13 7E A1 E6
[Jul 20 20:16:25 2020] INFO - Feedback auto error retrieval: device error code = 2620, error address = 4990, Modbus error = 3, error transaction id = 41446. Expected Modbus error of 3 and expected transaction id of 41446
[Jul 20 20:16:25 2020] ERROR - handle 2: First operation attempt received exception: 2620
[Jul 20 20:16:25 2020] TRACE - handle 2: Closing lowlevel conn type: 3, device type 7
[Jul 20 20:16:25 2020] TRACE - handle 2: Removing this user handle
[Jul 20 20:16:25 2020] TRACE - Creating NETWORK ListAll thread
[Jul 20 20:16:25 2020] TRACE - handle 3: Device constructor - connType: 2, identifier: 10.6.0.103, lowlevel connection on handle: 295
[Jul 20 20:16:25 2020] PACKET - handle 3: 12 byte send 0x A1 E8 00 00 00 06 01 4C
00 FD E8 01
[Jul 20 20:16:25 2020] PACKET - handle 3: 10 byte recv 0x A1 E8 00 00 00 04 01 4C
00 07
[Jul 20 20:16:25 2020] PACKET - handle 3: 12 byte send 0x A1 E9 00 00 00 06 01 4C
00 C1 C0 02
[Jul 20 20:16:25 2020] PACKET - handle 3: 12 byte recv 0x A1 E9 00 00 00 06 01 4C
00 00 00 0B
[Jul 20 20:16:25 2020] PACKET - handle 3: 12 byte send 0x A1 EA 00 00 00 06 01 4C
LJM_DEBUG_LOG_FILE (labjack.ljm.constants.DEBUG_LOG_FILE in Python) is a string based configuration, so use this method instead which is for reading string based configurations:
labjack.ljm.readLibraryConfigStringS(labjack.ljm.constants.DEBUG_LOG_FILE)
General LJM_DEBUG_LOG_FILE documentation is here:
https://labjack.com/support/software/api/ljm/constants/ljmdebuglogfile
labjack.ljm.readLibraryConfigS is value based (reads a float), and is not compatible with LJM_DEBUG_LOG_FILE.
What version of LJM and firmware are you running? Beta versions of both are usually pretty stable, so please try them:
Make sure you're running the latest software:
https://labjack.com/support/software/installers/ljm
Also, please update to the latest firmware using Kipling:
https://labjack.com/support/software/applications/t-series/kipling/devic...
Currently we use 1.0287 (which I believe is the latest) on the device itself but we use 1.1900 software because of (https://labjack.com/forums/t7/t7-terrible-streaming-performance-problems-latest-driver). Is there a generally available version of the software that fixes this performance issue or is that still in beta? I am a bit hesitant to use a beta as we use these in critical environments.
Just to follow up. I have updated to 1.21.0 beta and the issue seems to be resolved. Any chance you expect and official release of the driver for Linux any time soon?
LJM 1.21.0 can be considered official and stable.