Temporal Performance & Reliability

Reference Papers
Movie Performance
Benchmarking your System

Reference Papers

Data regarding the temporal precision of MonkeyLogic is available in the following publications. The first addresses the general issues encountered and the strategies employed in trying to achieve real-time control in a soft-real-time, interpreted-language operating environment. The second details specific performance characteristics of MonkeyLogic.

	Achieving behavioral control with millisecond resolution in a high-level programming environment. Wael F. Asaad & Emad N. Eskandar Journal of Neuroscience Methods, 2008, Vol.173, No.2, pp 235-240.
	A flexible software tool for temporally-precise behavioral control in Matlab. Wael F. Asaad & Emad N. Eskandar Journal of Neuroscience Methods, 2008, Vol. 174, pp. 245-258.

Addendum:

Resolution of certain timing inefficiencies (described in the second reference, in Table 1).

1) Calls to toggleobject that ended with updates of the control-screen, to plot or erase symbols reflecting the position of currently active visual stimuli, were noted to incur a large temporal cost (up to 30 ms). This issue has been resolved. The function "exit time" for toggleobject now averages less than 1 ms. The DrawMode:Fast option for toggleobject is now, therefore, unnecessary.

2) The first cycle in any behavioral tracking episode in eyejoytrack was noted to take much longer than subsequent ones due to the plotting of target rings around the relevant objects (second reference, section 5.3), up to nearly 30 ms. This issue has been resolved. Plotting the target rings now takes less than 2 ms, on average, so that the first tracking cycle latency is similar to subsequent cycle latencies when the eye- or joystick trace is updated.

3) The trial exit time (which averaged 10-20 ms) is now subtracted from the inter-trial-interval such that the actual ITI time now better approximates the desired ITI time.

Corrigendum:

In the second reference ("A flexible software tool..."), the example code for picking out the first instance of n consecutive correct trials from an error vector e (page 248) is incorrect. It should read:

f = find([1 logical(e) 1]);
t = min(f(diff(f) > n));

Note that the red "f" in the second line was miscoded as "find" in the published text.

Movie Performance

Since the publication of these papers, the ability to present movie stimuli (and translating images & movies) while simultaneously tracking behavior has been added, thanks to Markus Siegel, David Freedman, and Steve McClellan. As one would expect, the time required to update video frames does have a cost on the ability to rapidly sample behavior. Fortunately, this cost is relatively small. For example, shown here are the consecutive cycle latencies (i.e., time between behavioral data samples) while tracking eye movements with two movies running (and one of them simultaneously translating):

The red lines indicate those cycles in which a video update was performed to advance the frames (and position) of the movies. First, notice that no frames are missed (there is a red line every 10 ms for this case in which the video refresh rate was 100 Hz). Second, note that the latencies on those cycles in which a video update was performed took longer (between 2 and 3 ms) than those cycles without such video updates (~1 ms). This means that there are gaps in the on-line tracking of behavior equal to these times, potentially lasting up to nearly 3 ms. However, there is no temporal slippage; these gaps are simply intervals in which MonkeyLogic is temporarily "blind" to changes in the behavioral signals. The average latency across all cycles was 1.15 ms.

It is not unusual to have skipped frames during the first presentation of a movie in only the first trial, especially when using higher frame rates (this will be indicated by a warning message); however, we have found subsequent movie presentations in the following trials do not produce any skipped frames at rates of at least up to 100 Hz.

For a comparison of what this figure looks like when tracking behavior without concurrently showing movies, refer to Figure 3a of the second reference, above. In that figure, the periodic spikes in latency are less frequent, and due solely to the updating of the control-screen. The first cycle in that figure experiences a prolonged latency (~23 ms) due to updating the control-screen with symbolic representations of the visual stimuli; when presenting movies, this is automatically disabled such that there are no skipped frames (explaining why the longest latency here was under 3 ms).

This test was performed on the same machine used in the references above; much faster (and slower) computers are available, so performance will vary (see benchmarking, below).

Benchmarking your System

There are two ways built-in to MonkeyLogic to help benchmark a particular system (both found in the task panel of the main menu):

• MonkeyLogic Latency Test - This button will become enabled once a Conditions file has been loaded along with its settings. Using these settings, a benchmarking trial will be run to determine how quickly the behavioral signal can be sampled while presenting a static picture and while presenting a movie (the latter must perform video updates that momentarily interrupt behavioral sampling, as shown above). The measured times will vary as a function of the number and type of I/O assignments made (e.g., if no analog inputs are used, cycle latencies will be much shorter), and as a function of the number of analog input boards used (cycle latencies will be longer when two DAQ boards are used; nevertheless, this is preferred for reasons described in the first reference, above). The results will be plotted in a new figure window (as shown below). This figure is useful to determine the actual behavioral-sampling performance of one's system in the context of a real trial at the currently-selected I/O settings.

• Matlab Latency Test - This button simply time-stamps an empty-loop one million times and plots the resulting cycle latencies for each of the three process priorities available for use by MonkeyLogic (i.e., "Normal", "High" and "Highest."). This is useful to determine if there are sporadic high latencies caused by the operating system or background applications. Generally, mean latencies across priority settings will differ by a small amount, whereas maximum latencies will vary by several tenths of a millisecond or more. Shown below is an example of a resulting figure.