time stamping of streamed data

I also would like to know the time delay between the measurements of my three channels.  The data sheet says these are collected "as fast as possible" which isn't very informative.

See "Interchannel Delay" in Table A1.5:

https://labjack.com/support/datasheets/t7/appendix-a-1

So ideally I would be able to time stamp every measurement with a meaningful collection time based on the clock on the RPi, which is synchronized with the other RPi units.  Is this possible to do somehow?

At your scan rates of 1k-2k you could try low latency stream techniques.  Basically you tweak some settings that makes the T7 and LJM do minimal (or no) buffering and you read 1 scan at a time.  Thus you get each scan in pretty much real time which will typically be less than 1 ms delay.  Your data rates are at the upper end of streaming 1 at a time, though, so you might instead do a technique where you read every 100 ms and set buffer sizes so there is no leftover data each time you read.  This way you know the last scan has minimal delay and can timestamp the other scans based on the specified scan rate (which is very accurate). If you want to go down this route we can help further.

The scan clock on the T7 is quite accurate, so the typical techniques for synchronizing are addressing a couple things: 

1. Unknown about the delay between software telling stream to start and the first scan happening.

2. Clock difference between the T7 and the host over longer periods of time.  The T7 is 20 ppm at room temp (Table A5-6).

Some techniques:

A.  External scan clock.  Provide an external scan clock to the T7 so you are controlling when each scan happens.

B. Common signal.  Add another channel to the stream and include some other signal that can be used for synchronizing.  For example, perhaps 1 RPi produces a 1 pps pulse and this pulse goes to everything else so they can use it to synchronize in software.

the delay is 1335 microsec for resolution 8, but if I'm doing 1000 scans per second and each scan is 3 channels, then I need the delay between channels to be less than 1/3 ms.

The InterchannelDelay of 0.001335s suggests a max possible sample rate of 749 Hz.  Looking above at A.1.4 we specify a max sample rate of 630 Hz, so considering a little extra overhead and margin these numbers agree.  If you are scanning 3 channels at 1000 scans/second that is a sample rate of 3000 samples/second and definitely should not work.  Make sure you are setting STREAM_RESOLUTION_INDEX, not AIN#_RESOLUTION_INDEX or AIN_ALL_RESOLUTION_INDEX which only affect command-response mode. If you collect readings from a channel jumpered to GND you can compare to the noise levels from Table A.1.5 or A.3.1.1 to tell whether you are setting res index or not.

To further complicate, I just tested (LJM 1.1402 and T7 firmware 1.0208) and with ResIndex=8 it does not stream well at all.  I can get only get it to start around 30 samples/second or less and even then it does not run long before it has a problem. We are looking into that.

I initially collected data using resolution 9 and reading one at a time.  The time spacing I got when I did this exhibited a peculiar square wave shape,

In command-response the time spacing is dictated by the AIN Sample Time, your operating system, and your software.  From Table A.3.1.1 it takes about 3.5 ms/sample at ResIndex=9, so I could see writing software that holds a 5ms or 10ms loop pretty solid.

If I apply the "low latency stream techniques" is this similar to what I did already, reading data one value at a time with LJM_eReadName()?

Likely the stream data is being transferred (T7 buffer to LJM buffer) in large chunks in the background, even though your software is reading (from the LJM stream buffer) only 1 scan at a time.  With low-latency techniques you tweak advanced settings so data is transferred 1 scan at a time from device to host.  You can't transfer 100 ksamples/second this way, but in my experience you can do perhaps a few k.

  I'm wondering if I'd still have to deal with nonuniformity in the time spacing. 

This is just affecting the timing of when the data is transferred from device to LJM (and then you control when data is transferred from LJM to your software).  The actual acquisition is clocked in hardware by the stream clock and happens exactly at the known scan rate or does not happen at all.

What are the relative merits between methods A and B?  Does A require another piece of hardware to produce the clock, or can it come from the Pi? 

Both methods require a signal from something besides the T7.  What might work depends on what signals you have available or can produce.  I'm really not sure what sort of signal you can produce from a Pi itself.

The way we have things set up we were expecting to synchronize data collected on different Pi's in software after the data has been logged using recorded time stamps. 

Time stamps from where?  I'm not quite following but if you can clear me up a little I'll likely have more ideas.

Whenever I refer to sample rate I mean the number of scans of all three channels per second.

We would call that scan rate.  We define sample rate and scan rate in Appendix A-1:

https://labjack.com/support/datasheets/t7/appendix-a-1

Address - You can usually also call this a "channel".  An address usually returns the value of 1 channel.
Sample - A reading from one address.
Scan -  One reading from a list of addresses (scan list).

The scan rate, by definition, is a fraction of the sample rate where the fraction is the inverse of the number of channels being read in a single scan.  The scan rate is defined as:

ScanRate = SampleRate / NumAddresses

SampleRate = ScanRate * NumAddresses

My observed sample rate is variable, but is somewhere around 35 Hz.

From Table A.1.3 we expect a command-response reading to take about 3.5 ms/sample at Res=9.  If using eReadName (I don't see how eReadString applies) you are reading 1 at a time so add overhead from Table A.1.2 to each read.

I didn't try to control the sample rate from my program---this was what I got letting it run as fast as possible.  It's not clear to me how reliably I can control timing from the Pi.

C should have the ability to implement an interval timer in software that is based off the system clock of the computer.  Someone else here can follow up here if needed, but I would think we have some looping C examples that implement an interval timer (as opposed to a simple delay).  Basically you look at how long it takes for your execute and set the interval to something safely longer than that.

So you said above that "likely stream data is being transferred in the background..."  Does that refer to this situation?  In command and response mode does the labjack stream in the background?  That doesn't make sense to me.

No, does not apply.  When you do a command-response read there is really no buffering to think about.  You send the command and the device responds as soon as it can.

We decided to try the streaming.  You are correct that I did not set the resolution correctly.  I tried to figure out what the default resolution is and it says "0" on the modbus.  Is that the same as resolution 1?

The default ResolutionIndex=0 equates to:

T7 = 8 in command-response and 1 in stream.

T7-Pro = 9 in command-response and 1 in stream.

I'm a little puzzled now because the manual suggests I should be able to stream much faster if I'm at resolution 1.  When I push up above 2315 Hz scan rate I get an error about overlap and my program fails.

If you set Res=0 and every channel in the scan list is set to Range=10 you should be able to get a sample rate of 100k or more.  Also assumes STREAM_SETTLING_US=0.

Add a write of AIN_ALL_RANGE=10 (or =0) before you start the stream to make sure all channels are using the +/-10V range.

If we use low latency streaming does that mean that data will get to the Pi at a time close to when it was collected, so I could use that as a way of applying time synchronization?

Exactly.  In my experience the time between acquisition and getting the scan in software roughly half a millisecond.  That is what we mean by low-latency.

Do I understand correctly that with low latency streaming the data collection occurs at fixed intervals chosen by the labjack and I fetch data one scan at a time?

The former is always true in stream.  The latter, host app reads 1 scan at a time, is a typical part of low-latency streaming technique.  You can read 1 scan at a time with any stream, but the detail is that without doing some advanced stream configuration the data will be moved device to LJM in large chunks in the background:

https://labjack.com/support/software/api/ljm/function-reference/ljmestre...

If the labjack's sample spacing is reliable, may not be worth the trouble...unless it's the only way to determine when the first data value was collected.

The spacing is very reliable, per the LabJack's clock, then you are exactly right that the main issue is knowing the exact time the stream starts in relation to the host's clock.

limiting the duration of the data collection to some duration longer than a minute, so I could start with a time stamp from the Pi's system clock and assume the labjack's specified sample rate from there to apply time stamps going forward.

Right again.  If the T7 clock is 20 ppm fast, and the host clock is 20 ppm slow, that means they differ by 40 ppm which is 40us per second, or 1 millisecond every 25 seconds.

It seems like for this to work I just need to be able to relate the stream start time to the Pi's clock to within a millisecond. 

Yes, you've got it.

I have a new idea that should work within 1 ms using normal easy stream.

1.  We will add a new register that records the value of CORE_TIMER at the start of the first scan of stream.  From the JSON:

{"address":61520, "name":"CORE_TIMER", "type":"UINT32", "devices":[{"device":"T7", "fwmin":1.0071}], "readwrite":"R", "tags":["CONFIG"], "streamable":true, "description":"Internal 32-bit system timer running at 1/2 core speed, thus normally 80M/2 => 40 MHz."},

Note from Section 3.2 that right now you can actually add a couple channels to your stream and record CORE_TIMER with your stream data.  This is really overkill, though, as you just need to know the time of 1 scan as all others will have a relative time of exactly ScanNumber/ScanRate.

2.  Now you just need to relate CORE_TIMER to the time on your host.  You should be able to do this within +/-1ms at any time by doing a command-response read of CORE_TIMER while at the same time noting the host's system clock.  Per the rough clock drift math above I would do this once per minute.

I measured the time that the call to LJM_eReadString requires when resolution=9 and it varies from 6.65 ms to 10.3 ms.  If I assume that the time to associate with the data is the time when the call returns (which is what I did assume) then I can't get uniform sampling this way because the call duration is not predictable.  Note also that this is a lot more than the 3.5 ms/sample quoted on table A1.3. 

If those times are accurate for your situation, I would suggest a software timed loop interval of 20 ms so you know the work can be safely done each time through the loop and you will get regular timing.

Still not sure what you are doing with eReadString. I would expect eReadName, or for best speed eReadAddress.  And if doing multiple channels you would use eReadNames or eReadAddresses so you only have 1 overhead from Table A.1.2.

In LabVIEW I have an example "C-R Speed Test.vi" to check command-response speeds.  Do you see something similar in our C examples?

I tested over USB with #Iterations=128, Range=10, and ResIndex=9:

1 Channel = 4.3 ms/iteration
2 Channels = 7.8 ms/iteration
3 Channels = 11.4 ms/iteration

So I have an overhead of about 0.8 ms plus about 3.5 ms/sample.

I don't understand how a software timed loop will help.

The software timed loop is useful if you use command-response mode and want regular scan intervals.  You could make a loop that executes every 20 ms and each time scans your 3 channels.

I have a vague recollection of using eReadNames and it wasn't faster, but I'll revisit that. 

In the test I ran it was doing 128 calls to eReadAddresses where each call read 3 channels, and this took about 0.8 + 3*3.5 ms.  If instead I made 3 calls to eReadAddress each time it would take about 3 * (0.8 + 3.5) ms.

If I used eReadName or Names intead, the 3.5 would be a little longer but not much.

  You are right, I'm using eReadName.   The test code looks something like:

It is tough to measure down to a few milliseconds with the normal windows timers.  Instead you should do a bunch of iterations so it takes perhaps a second.  For testing to determine the time per iteration I suggest code like:

start=get_time();
For 512 iterations;
    err = LJM_eReadNames(...);
end = get_time();
timeperiteration = (end-start)/512

Or to test top performance do:

NamesToAddresses()
start=get_time();
For 512 iterations;
    err = LJM_eReadAddresses(...);
end = get_time();
timeperiteration = (end-start)/512

Hard to say what sort of noise might be on your signal, so first test with a channel jumpered to GND (or AIN15).  When I watch the readings from AIN15 with Range = 10 and Res = 0 or 1, I see 5 different values: -0.000027, 0.000289, 0.000604, 0.000920, and 0.001235.  When I change Res = 3 I pretty much just see 2 values.  And Res = 5+ it pretty much just sits on 1 value.

To add, in our C/C++ examples there is a testing/c-r_speed_test.c example for testing command-response speeds on your system. This is basically the C/C++ version of the "C-R Speed Test.vi" example we mentioned earlier.

Is LJM_eReadName much slower than LJM_eReadAddress?

Not too much.  Maybe 10%.

The software timed loop is only useful if I'm doing the timing incorrectly.  I'm setting a time stamp for a sample when the call to LJM_eReadName returns.

That all sounds correct, but without an interval loop you wind up with scans that are not evenly spaced.  They have the correct timestamp but are not evenly spaced.  That is what you could fix by adding an interval timer.  If you know the scans typically take 10-15 ms to complete, then put them in a loop that executes every 20 ms and you will get a scan every 20 ms.

Are all the GND terminals internally connected together?

Yes.  All GND terminals are the same thing.

Without the new register that records the scan time start am I right that the only way to get the time sync I'm looking for is to stream CORE_TIMER?

Another possibility might be to read core-timer right before you start the stream, and then add a constant that reflects the typical time from when you call startstream to when the first scan actually starts.  I just checked this on my machine (LJM 1.1402 and firmware 1.0208) and the time was 43-46 ms.  Beyond the 1 ms variability you want, but still pretty consistent.

I was wondering about whether I could do it with just the lower 16 bits if I check the full CORE_TIMER just before or after starting the stream, so it's guaranteed to be less than 65 ms before the scan starts.

Since the core-timer is 40 MHz it will roll the lower 16-bits every 1.6 ms, so it will be tough to use just the lower 16 bits.

I thought I read somewhere that while streaming you couldn't do anything else because the labjack was very busy.  So this is not the case?

That is true for the U12, but all our other devices can do other things during stream.  The device is busy while streaming and stream can impact other things, but in general you can do about anything via command-response except analog input reads.

How long would it take for you to add the new register?  (Is that a library change?  Firmware on the labjack?  Both?)

It is just a firmware change to add the new register.  You could then read it by address.  To read it by name you would need a new JSON that defines the new name for LJM.  Usually we can get something like this done very quick, day or 2, but we are focused on a different firmware project at the moment and might not be able to jump on this very quickly.

If I measure time after the LJ_eReadName call returns, and this call takes a variable amount of time, then there's no way to achieve uniformity.  I can make the calls start at a uniform time but I can't make them finish at a uniform time. It sounds like you're saying that this call will run in a uniform amount of time if I call it less frequently, which doesn't match up with my idea of how command and response works.  I thought you issue command and get response, and so there should be no advantage in waiting between commands, because the labjack is done with the last command and is idle at that point.  Do I have a misconception here?

If you use an interval timed loop the average time per loop will be spot on your specified interval, but you are correct that and variability in the execution time of each command will be jitter.

I added an interval timing feature to measure the execution time of each command in my eReadNames C-R speed test program.  Resolution is only 1 ms but works decent here.  I did 1000 iterations of 3 channels at Range=10 and ResIndex=9.  The average time per iteration was 12.4 ms.  One iteration took 17 ms, 2 took 16 ms, 2 took 15 ms, and the rest took 10-14 ms.

My implementation then would be to make a 20 ms interval loop that calls this eReadNames command.  For my timestamp I would read the system timer at the beginning of each loop so the timestamp always increments by 20 ms.  Thus it is a timestamp of when made each call.  The variability of the time per call means that there will be perhaps +/-3.5 ms jitter in the actual time of each call, but over the course of 10 seconds I will get 500 calls.

There is the other question of why my data rate is much lower than the spec when I run this way.
Start by running the test programs mentioned in post #12 to see what results you get.  I would have to ask someone else here if a Pi typically hits the same speeds we see on full machines.


I thought CORE_TIMER was reporting in microseconds.  I was trying to infer its units from my measurements, though, since I couldn't find any documentation, so I must have gotten confused.   If it's 40 MHz that's obviously going to make the higher 16 bits more important.
Here is the description from the JSON:

"Internal 32-bit system timer running at 1/2 core speed, thus normally 80M/2 => 40 MHz."

I've been trying to estimate this delay value using code that looks like:
LJM_eStreamStart(...);
LJM_eReadAddress(  ... CORE_TIMER ...);
get_system_clock();
When I do this at a scan rate of 160 Hz (with 5 channels, my 3 data channels and 2 clock channels) this delay is typically around 4 ms.  When I do it at a faster sample rate the value is sometimes negative.

I'm not quite following.  You call streamstart, read the core-timer, and then note your system clock.  What delay is 4ms or negative?

any advise on how to determine the maximum reasonable streaming rate?  And how to avoid hanging and errors? 

The hanging should not happen, but could be related to your scan rate problems so lets tackle that.

- Make sure you are running the latest LJM and firmware and confirm to us what you are running.  Someone here can test the same using our Python stream example on a Pi.  We are running LJM 1.1402 and T7 firmware 1.0208 ourselves, and on my Windows machine I did notice a problem with ResIndex>1 streams (not going nearly as fast as they should) but at ResIndex=1 I can go 3x35k over USB and even faster over Ethernet.

- In addition to your write of STREAM_RESOLUTION_INDEX=n, add a couple writes before your streamstart to do AIN_ALL_RANGE=10 and STREAM_SETTLING_US=0 so we know for sure what range and settling are set to.

Another question: when streaming what is the expected delay between channels of the same scan?  Should it be the value from the last column of table A.1.3?

Those would give a rough guess, but the actual number you want for stream mode is the "Interchannel Channel" delay specified in the last column of Table A.1.5.

https://labjack.com/support/datasheets/t7/appendix-a-1

Regarding max. stream speeds tests on our Raspberry Pi 3 using LJM 1.1200 (latest beta armhf build) and T7 firmware 1.0208 (latest beta):

With 5 channels with resolution index 1, around 13 to 14K I saw reliable stream scan rates speeds. I tested in C since that is what you are using and used a USB connection. Note that by "reliable" I mean there were no -9999.0 samples which are skipped samples caused by a stream buffer overflow error on the T7. I could get up to around 23K but with skipped samples, then eventually run into STREAM_SCAN_RATE_OVERLAP and STREAM_SCAN_RATE_INVALID errors.

With 3 channels and resolution index 1, reliable scan rate speeds are around 19 K. Over 19K to 24K, after starting stream mode there is an initial stream buffer overflow (-9990.0 samples), but after this initial overflow the stream was running fine until the next stream start.

Try the latest T7 beta firmware and see if that helps the INCORRECT_NUM_RESPONSE_BYTES_RECEIVED errors. If not, then we may have resolved that further with a newer LJM version, and would need to look into updating the ARMv7 build. STREAM_SCAN_RATE_OVERLAP and STREAM_SCAN_RATE_INVALID errors will indicate that the T7 hardware cannot stream faster with current stream configuration.

Relating to maximum speeds and trying to improve performance, try starting your application's process with a higher CPU priority and see if that helps software stream performance. This might help with the -9999.0 samples and LJME_LJM_BUFFER_FULL errors at faster stream speeds. You can use the "nice" command to start an application and specify its priority. I will point out that it seems like you are using an older Raspberry Pi with a slower processor, so in your tests you might not be able to reach the stable/reliable stream speeds I was seeing. What Raspberry Pi model are you using?

With the STREAM_IS_ACTIVE error, that indicates that stream mode was started but not stopped. In this case power cycling the T7 will stop stream mode, or you can detect this error, stop stream mode and then start stream mode again.

We are looking further into the other issues.

We are working on new firmware to solve this issue and add the new register that will report the stream_start time stamp. Please send an email to [email protected] so that we can send you the new firmware when it is ready.

For testing, power cycle your T7, then in your application open the T7 and then do eStreamStart with minimal calls in between. Is there a hang? To double check, by "hang" do you mean that the eStreamStart call stalls indefinitely and does not return?

If you find that certain calls/configuration before eStreamStart are causing issues, please provide the calls.

For your test I meant remove the calls before eStreamStart and see if you still have the problem. This is to see if a certain configuration or set of calls are causing an issue, or if stream mode in general is having the issue. Also, you could try our stream_example.c example in the C/C++ examples download and see if you get hangs.

If you find a set of calls that can cause the issue, please provide that chunk of code so we can see the registers you are reading/writing and how your eStream start is configured so we can try to reproduce the issue.

I wasn't able to reproduce the hang on LJM_eStreamStart yet. I will continue looking into this tomorrow and I will update this thread tomorrow as well.

I tried again to reproduce the LJM_eStreamStart hang, but was unable to. I compiled and ran your program many times on a Pi 3 rev 4. I tried altering your program:

• Tried uncommenting the call to LJM_eStreamRead
• Tried uncommenting the call to setresolution
• Tried re-commenting the calls

I tried with LJM 1.1403 firmware 1.0214 via USB connection.

Some ideas:

• Try updating to the newest LJM version (1.1403) for armhf (currently beta): https://labjack.com/support/software/installers/ljm
• Try disabling WiFi. (Write 0 to POWER_WIFI_DEFAULT)
• Try reverting to factory settings using Kipling
• Try using gdb to see what's going on:

gdb ljtest
(gdb) r
<wait until it's hanging, then press ctrl + c>
(gdb) thread apply all bt

Then, let us know what the output of `thread apply all bt` is. The LJM .so is stripped (of debug symbols), but in my quick test I was able to see some LJM functions in the backtrace.

What's the longest you've left it to hang for? (Tens of seconds / minutes / tens of minutes?)

12 minutes is long enough to indicate that there's almost definitely a deadlock, rather than any sort of delay. Try the armhf version. It's for ARMv7 with hard floats. (We probably need to rename our release types in the future to help avoid confusion.) The recent 1.1403 release might fix some old bug that causes a deadlock. Let us know if it doesn't work, of course.

LJM does handle signals (by default). If you are currently handling signals, I recommend either not handling them or calling the LJM handler from your handler as described here:

https://labjack.com/support/software/api/ljm/does-ljm-handle-signals

I've replicated the -9999 issue. I'm not sure what's causing it, but I'll keep you updated. One thing that I noticed is that the initial -9999 error never happened for the first stream after the T7 was unplugged and replugged. Alternately, I was able prevent the initial -9999 by rebooting the device by writing 0x4C4A0000 to SYSTEM_REBOOT after stream. See the attached stream_example.c.

I'll update this thread when I have more information.