It is rather common that when we turn on our computer, we immediately look at the screen waiting for its response. As simple as this may appear, such eye movements towards locations at which we expect consequences of our actions inform us about how we control our actions. In the laboratory, using eye trackers with high temporal and spatial resolution, such anticipatory eye movements have been studied for several years (e.g., Gouret & Pfeuffer, 2021; Pfeuffer et al., 2016, 2022). Here, we assessed whether and under what conditions anticipatory eye movement measures relevant for the study of human action control could reliably be replicated online using participants’ own webcams. A reliable assessment of anticipatory eye movements even under suboptimal tracking conditions, for instance, online, is also a precondition for their future usage in the context of human-technology interaction settings.
Ideomotor theory (Elsner et al., 2002; Elsner & Hommel, 2001, 2004; Hommel, 2009; Hommel et al., 2001; James 1890/1981; Kunde, 2001; Sun et al., 2020; see e.g., Shin et al., 2010, for a review) postulates that when an action contingently yields the same effect, a bi-directional action-effect association is formed between action and effect (e.g., Dutzi & Hommel, 2009, Elsner & Hommel, 2001; Hommel et al., 2001; Kunde, 2001). Evidence for this notion comes, for instance, from the finding that participants respond faster when their key presses contingently produce spatially response-effect compatible visual effects (left key press –> visual effect on the left; e.g., 1st half of the experiment) than when their actions produce spatially response-effect incompatible visual effects (left key press –> visual effect on the right; e.g., 2nd half of the experiment). Such response-effect compatibility effects demonstrate that we anticipate our actions’ effects before executing the corresponding action (see, e.g., Hoffmann, 2003; Koch & Kunde, 2002; Pfister et al., 2010; 2014; Pfister, Janczyk, et al., 2014; Wirth et al., 2015). This effect anticipation allows us to retrieve the associated action and thus select the appropriate action to realize the intended effect (strong version of ideomotor theory; e.g., Elsner & Hommel, 2001; Hommel et al., 2001; Kunde, 2001).
Importantly, anticipating our actions’ effects does not only allow us to select appropriate actions that will produce these effects. It also starts a process of proactive effect monitoring in preparation for later comparing expected and actual effect as evident in anticipatory saccades (e.g., Gouret & Pfeuffer, 2021; Pfeuffer et al., 2016, 2022). When the location of visual effects is predictable due to prior learning experiences, in a blank screen interval between a manual response target offset and effect onset, participants look more often towards the future effect’s location than in the opposite direction (saccade-effect congruency, SEC, effect; e.g., Gouret & Pfeuffer, 2021; Pfeuffer et al., 2016; 2022). This was the case both when action-effect contingencies persisted for a large number of trials (1st vs. 2nd half of the experiment; e.g., Pfeuffer et al., 2016, 2022) and when action-effect contingencies only persisted for sequences of several trials before switching, that is, after few prior action-effect learning instances (Gouret & Pfeuffer, 2021).
Interestingly, participants do not only anticipate the effects’ future location but also the time at which their actions’ effects will appear. When effect delays were long rather than short, participants responded slower (Dignath et al., 2014, 2017). This has been taken as evidence that anticipating a further delayed effect takes longer, that is, the timing of effects is included in the action-effect association as well as the effect anticipation. Specifically, participants also anticipated when the consequences of their actions would occur regardless of what these consequences were (Dignath & Janczyk, 2017).
The timing of future effects is especially important when one tries to proactively monitor effects. Correspondingly, it has been shown that anticipatory saccades do not only reflect the location but also the timing of anticipated future effects (Gouret & Pfeuffer, under review/preprint; see also Kenward, 2010, for similar prior findings in infants). When left/right manual responses predictably caused visual effects after a short/long effect delay, participants’ first saccade towards the future effect’s location (effect-congruent) was performed earlier/later corresponding to the effect delay. This was the case even when action-effect delay mappings were only consistent within short trial sequences (Gouret & Pfeuffer, under review/preprint). That is, anticipatory saccade latencies (i.e., proactive effect monitoring) were adapted to the timing of future effects after few learning instances.
Importantly, processes of proactive effect monitoring and corresponding anticipatory saccades, so far, have only been studied with in-laboratory eye trackers with high temporo-spatial resolution. For an application to a larger range of studies and to confirm the reliability of anticipatory saccades as a measure of proactive effect monitoring under suboptimal eye tracking conditions, it is essential to assess to what degree anticipatory saccades can also be observed with eye trackers with low temporo-spatial resolution. Especially, demonstrating that anticipatory saccade frequency and latency effects can also reliably be measured with online eye trackers using participants’ own webcams is interesting for a future usage in applied contexts. For instance, anticipatory saccades occurring during the usage of various interfaces assessed via eye tracking based on the users own webcam could provide new insights into human-technology interaction or even be used as pieces of information guiding interface responses.
Regarding behavioural experiments, several comparisons of in-laboratory and online experiments have already been conducted (Del Popolo et al., 2022; Gosling et al., 2004; Innes et al., 2020; Reimers & Stewart, 2015; Semmelmann & Weigelt, 2016). There have also been first assessments of online eye tracking using participants own webcams as compared to in-laboratory experiments (e.g., Johansen et al., 2011; Semmelmann & Weigelt, 2018). Such prior studies demonstrated that online behavioural as well as online eye tracking experiments using participants’ own webcam provide rather reliable data. Here, we assessed whether anticipatory saccades occurring during goal-directed action control could also be reliably assessed via participants’ personal computers and webcams. Furthermore, we assessed the impact of effect relevance on anticipatory saccades to further understand the mechanisms of proactive effect monitoring in human action control.
To generally confirm the reliability of anticipatory saccade frequency and latency effects using webcam-based eye tracking of relatively low temporo-spatial resolution, we conducted two online eye tracking experiments (building on the paradigm of Pfeuffer and colleagues (Gouret & Pfeuffer, 2021; Pfeuffer et al., 2016, 2022). In both experiments, participants had to respond to a forced choice, response repeat/switch target with a left/right keypress and correct responses predictably led to a visual effect on the left/right side of the screen (i.e., at a spatially response-effect compatible/incompatible location; 1st vs. 2nd half of the experiment). Furthermore, one response caused an effect after a short delay, whereas the other response caused an effect after a long delay. We used participants’ webcams to assess their eye movements between target offset and effect onset. In Experiment 1, like in prior studies (e.g., Gouret & Pfeuffer, 2021; Pfeuffer et al., 2016, 2022), effects were irrelevant. In Experiment 2, extending prior in-laboratory studies, we rendered effects relevant to investigate the impact of effect relevance on anticipatory saccades in the online eye tracking setting.
We hypothesized that we would be able to replicate both the SEC effect in anticipatory saccade frequencies (e.g., Gouret & Pfeuffer, 2021; Pfeuffer et al., 2016, 2022) as well as the influence of effect delay on anticipatory saccade latencies (Gouret & Pfeuffer; under review/preprint) online using participants’ own webcams. Furthermore, based on the findings of Experiment 1 using irrelevant effects, we presumed that effect relevance would boost both frequency and latency effects in anticipatory saccades.
In Experiment 1, participants’ actions produced predictable but irrelevant visual effects.
The central effect we wished to replicate online in Experiment 1 was the influence of effect delay on saccade latencies. We simulated the sample size required to observe an influence of effect delay (200 ms vs. 800 ms) on saccade latencies based on a corresponding linear mixed model of the data of Gouret and Pfeuffer (under review/preprint) as suggested by Kumle et al. (2021). This sample size simulation suggested that ten participants would be required to observe an influence of effect delay on saccade latencies with at least 80% power at α = .05. To enable future comparisons with our prior experiments and to account for a potentially reduced effect in an online setting, we decided upon a planned sample size of roughly 24 participants. Note that the difference between the observed frequency of anticipatory saccades towards the future effect rather than in the opposite direction as compared to chance has consistently been observed to show large effects (e.g., Gouret & Pfeuffer, 2021; Pfeuffer et al., 2016, 2022) that a sample of 24 participants is sufficient to replicate as well.
Twenty-four participants completed the online eye tracking experiment (6 males, 18 females, mean age = 34.1 years, SD = 9.8, 3 left-handed, 1 Psychology student, 23 workers). Participants were recruited via Prolific and were native English speakers, had normal or corrected-to-normal vision, and a Prolific approval rate of at least 95%. Only participants who had a computer or laptop with a screen of at least 15” as well as a webcam could join the experiment. Furthermore, the data of participants who aborted the experiment, who switched tabs during the task, whose sampling rate was below 25 samples/second or who could not be tracked with at least 80% accuracy in an initial tracking check, or who performed at less than 70% accuracy in an initial practice (see Design and Procedure) was not considered. For these participants, the experiment was automatically aborted and they were replaced with new participants. Moreover, participants whose mean error rates exceeded the sample average by more than three standard deviations were also excluded and replaced. No participant was excluded due to this criterion. All participants were naïve to the purpose of the experiment, provided informed consent at the beginning of the online experiment, and received 7.50 GBP as compensation for their participation. Participants who could not complete the experiment received compensation in accordance with Prolific guidelines. Thirteen additional participants started the experiment, but failed to meet the technical requirements (e.g., low sampling rate or insufficient tracking accuracy in a first check). For these participants, the experiment was aborted with a corresponding notification and they were compensated for the time they spent. The study was conducted in agreement with the Declaration of Helsinki and the guidelines set by the local ethics committee.
Participants took part at home on their own computer or laptop. The experiment was programmed in JsPsych 6.3.1 and eye movements were tracked using the Webgazer library (Papoutsaki et al., 2017) and participants’ personal webcam. At the beginning of the experiment, participants used a bank card to resize a rectangle on the screen, so that stimulus size was the same for all participants. Moreover, an initial tracking check assessed the sampling rate and the accuracy with which participants’ eye movements could be tracked. Throughout the experiment, before each block, 9 points calibration and validation were performed and repeated up to 5 times in case tracking accuracy did not reach at least 80%. The index finger of participants’ left and right hand rested on the keys S and L. Throughout the experiment, the screen background was black.
Participants were first asked a set of demographic questions. Then, they conducted a practice of the task (20 trials without effects) in which participants needed to reach 70% accuracy (one retry possible). Subsequently, the main experiment consisting of 8 blocks of 50 trials ensued and was followed by post-experiment questions regarding encountered problems or disturbances as well as the presumed purpose of the study and observed regularities. Participants’ eye movements and performance measures were only recorded during the main experiment.
Each trial of the main experiment started with the presentation of a white fixation cross (0.6°) in the middle of the screen for a jittered inter-trial interval (ITI) ranging between 1000 ms and 1500 ms. The fixation cross was then replaced by a target (100 ms, 0.6°) indicating whether to press the left (S) or right (L) key. For every first trial of a block, the target was an arrow pointing either to the left or to the right. The direction indicated the first key to press. In the other trials of each block, the target was either an “=”, indicating to press the same key as in the previous trial (same response/response repeat), or a “x” indicating to press the opposite key as compared to the previous trial (opposite key/response switch). The reference was always the correct response on the previous trial. That is, after an error, participants had to select the same/opposite response according to the response they had not executed on the previous trial. These response repeat/switch forced choice targets were chosen to ensure that only action-effect associations and not target-effect associations could influence effect anticipation and corresponding anticipatory saccades (see Pfeuffer et al., 2022, for evidence that anticipatory saccade measures do not differ between left/right and response repeat/switch forced choice targets). The target was followed by a blank screen response frame of up to 1400 ms (i.e., response limit 1500 ms; see Figure 1 for the trial structure). Target onset was set as the zero mark for both manual reaction times and saccade latencies even though only saccades starting after target offset were analysed.
Correct responses were followed by a blank screen action-effect interval lasting for 200 ms or 800 ms (effect delay; e.g., left response – 800 ms, right response – 200 ms). After the action-effect interval, a visual effect (orange vs. blue circle, 500 ms, 1.5°) appeared 14° to the left/right of the screen centre. Per participant, one response was contingently associated with one effect colour and effect delay and the mappings were counterbalanced across participants. In one half of the experiment (e.g., blocks 1–4), effects appeared at locations spatially response-effect compatible with the manual response (e.g., left response –> effect on the left), in the other half of the experiment (e.g., blocks 5–8), effects appeared at locations spatially response-effect incompatible with the manual response (e.g., left response –> effect on the right). Response-effect compatibility order was counterbalanced across participants. After the effect was presented, the next trial began.
In case of an incorrect, premature, or omitted response, no effect was presented. Instead corresponding feedback was displayed in red in the centre of the screen (“too early!” for premature responses, “error!” for incorrect responses, and “too slow!” for response omissions; duration: 1000 ms) and the trial was aborted.
At the end of each block of the main experiment, participants received feedback about their performance (i.e., number of errors, omitted or premature responses). They were then reminded to respond as fast and accurately as possible.
Participants were not informed about contingencies between their responses and effect delays, effect positions, or effect colours. Moreover, no instruction or information regarding eye movements was given. Thus, any saccade performed during the assessed anticipatory interval between the target offset and effect onset can be considered as spontaneous and uninstructed.
All statistical analyses were performed using R version 4.1.2. Practice trials, the first trial per block as well as trials with premature (<0.1%) or omitted responses (<0.1%) were excluded from all analyses. Linear mixed models (LMMs) were used to assess saccade latency and manual reaction time and generalized linear mixed model (GLMMs) were used to assess saccade-effect congruency and errors. (G)LMMs were computed with the lme4 (Bates et al., 2015), lmerTest (Kuznetsova et al., 2017), pbkrtest (Halekoh & Højsgaard, 2014), afex (Singmann et al., 2015), and emmeans (Lenth et al., 2018) packages. To create figures, we used the ggplot2 package (Wickham, 2016). For LMMs, we used the maximum likelihood estimation and the Satterthwaite (Kuznetsova et al., 2017) method to assess p values for model selection and restricted maximum likelihood estimation and the Kenward-Roger (Kenward & Roger, 1997) approximation for denominator degrees of freedom to examine the final model. For GLMMs, we used maximum likelihood estimation and binomial link functions (bobyqa optimizer, 1,000,000 iterations) and p values were estimated via asymptotic Wald tests. For GLMMs, we report odds ratios (Szumilas, 2010). Conditional R2 values for each mixed model were computed using the r.squaredGLMM function of the MuMIn package (Nakagawa & Schielzeth, 2013; see also Nakagawa et al., 2017). Both of our two-level predictors were contrast coded (–1/1) such that estimates in the linear mixed model table (β) indicate the difference between one of the conditions and the grand mean.
The (G)LMMs of all dependent variables included the fixed effects response-effect compatibility (response-effect compatible vs. response-effect incompatible) and effect delay (short vs. long) as well as their interaction. For all dependent variables, we started with a random effects structure that included participant intercepts and by-participant random slopes for response-effect compatibility and effect delay. In case the (G)LMM did not converge, the model’s complexity was decreased. We first removed correlations among random slopes, then the by-participant random slopes of effects response-effect compatibility, and finally the by-participant random slopes of effect delay until the model converged without a singular fit or a negative Hessian eigenvalue.
For the analysis of manual reaction times (RTs), trials containing errors as well as trials with an RT deviating more than three SDs from their individual cell mean (i.e., outliers) were not considered. The mean RT was 494 ms (SD = 149 ms). The LMM fitting the manual RT included participant intercepts and by-participant random slopes of effect delay as random effects (see Figure 2B). No significant effect was observed, |ts| ≤ 1.40, ps ≥ .174 (see Table 1 for detailed results).
|MANUAL REACTION TIME (RT)|
|EXPERIMENTS||EXPERIMENT 1||EXPERIMENT 2|
|Response-effect compatibility × effect delay||0.43||–2.32–3.18||1.40||0.31||0.759||1.36||–1.58–4.30||1.50||0.91||0.364|
|Marginal R2/Conditional R2||0.000/0.222||0.002/0.267|
The mean error rate was 3.8% (SD = 2.1%). The GLMM fitting error (0 = correct, 1 = error) included participant intercepts and by-participant random slopes for response-effect compatibility and effect delay as random effect (see Figure 2A). No significant effect was observed, |ts| ≤ 1.23, ps ≥ .219 (see Table 2 for detailed results).
|EXPERIMENTS||EXPERIMENT 1||EXPERIMENT 2|
|Intercept||0.03||0.03 – 0.04||0.00||–28.40||<0.001||0.04||0.03 – 0.05||0.01||–21.73||<0.001|
|Response-effect compatibility||0.91||0.79 – 1.06||0.07||–1.23||0.219||0.96||0.84 – 1.11||0.07||–0.50||0.616|
|effect delay||0.93||0.80 – 1.07||0.07||–1.01||0.312||0.95||0.83 – 1.09||0.07||–0.76||0.450|
|Response-effect compatibility × effect delay||1.05||0.94 – 1.17||0.06||0.86||0.388||0.97||0.88 – 1.07||0.05||–0.58||0.563|
|Marginal R2/Conditional R2||0.005/0.047||0.001/0.056|
Only saccades occurring during the anticipatory interval between target offset and effect onset were assessed. In the present study, we determined saccade starting points and latencies by assessing between which consecutive samples the horizontal gaze position shifted by at least 2°. The data of the pre-movement sample was set as the saccade’s starting point and latency. Only the first saccade per trials was considered for the analysis of saccade-effect congruency and only the first effect-congruent saccade (i.e., directed towards the future effect) was considered for the analysis of saccade latencies. Note that this differs from prior in-laboratory studies on such anticipatory saccades in which all saccades per trial were considered for the analysis of saccade-effect congruency. The criteria for analysis were adapted to account for the temporally less precise saccade detection using participants’ individual webcams which sample at only around 30 Hz at maximum under ideal Browser conditions. In sum, 3,516 saccades were used for the saccade-effect congruency analysis and 2,791 saccades were considered for the saccade latency analysis. To infer tracking precision, we additionally assessed gaze sample distributions during target presentation. The mean dispersion of gaze samples in X direction during target presentation was 1.1° and the mean dispersion of gaze samples in Y direction was 1.6°. That is, gaze sample dispersion in a phase of the experiment in which participants should fixate one spot, the target location, was lower than the 2° saccade criterion we set to detect saccades.
The overall SEC score was computed dividing the number of first saccades per trial towards the future effect location (effect-congruent) by all saccades (including both effect-congruent and effect-incongruent saccades). A score higher than 50% (chance level) shows that participants anticipated their own actions’ consequences and moved their eyes towards them in anticipation, that is, proactively monitored their actions’ effects.
We thus, first conducted two one-sample t-tests to assess participants’ overall SEC scores. These one-sample t-tests showed that participants’ mean SEC was significantly greater than 50% in the response-effect compatible condition, t(23) = 3.12, p = .002, d = 0.64 (M = 55.8%, SD = 9.1%), but not in the response-effect incompatible condition, t(23) = 1.04, p = .155 (M = 52.4%, SD = 11.1%).
The GLMM fitting saccade-effect congruency (0 = effect-incongruent, 1 = effect-congruent) included response-effect compatibility and effect delay slopes and participant intercepts as random effects (see Figure 3A). Response-effect compatibility reached significance, z = 3.51, p < .001, OR = 2.26. That is, participants performed more effect-congruent saccades when the future effect’s location was spatially response-effect compatible rather than response-effect incompatible to their response. Neither effect delay nor the interaction of response-effect compatibility and effect delay was significant, |zs| ≤ 0.88, ps ≥ .380 (see Table 3).
|SACCADE EFFECT CONGRUENCY|
|EXPERIMENTS||EXPERIMENT 1||EXPERIMENT 2|
|Intercept||1.13||1.02 – 1.25||0.06||2.39||0.017||1.75||1.51 – 2.04||0.14||7.25||<0.001|
|Response-effect compatibility||1.13||1.06 – 1.21||0.04||3.51||<0.001||1.48||1.27 – 1.73||0.12||5.01||<0.001|
|effect delay||0.99||0.93 – 1.06||0.03||–0.21||0.837||1.01||0.91 – 1.13||0.06||0.17||0.864|
|Response-effect compatibility × effect delay||0.97||0.91 – 1.04||0.03||–0.88||0.380||1.1||1.02 – 1.17||0.04||2.66||0.008|
|Marginal R2/Conditional R2||0.005/0.015||0.043/0.091|
For the saccade latency analysis, we only considered the first effect-congruent saccade per trial. The LMM on saccade latency included participant intercepts and by-participant random slopes for effect delay as random effects (see Figure 3B). The response-effect compatibility did not reach significance, t = –1.83, p = .080, |2β1| = 37.82 ms. Effect delay reached significance, t = –12.11, p < .001, |2β| = 220 ms. That is, participants’ first effect-congruent saccade was performed later for the long effect delay than for the short effect delay. Furthermore, effect delay and response-effect compatibility significantly interacted, t = 2.44, p = .015, |2β| = 28 ms (see Table 4). Participants showed larger latency differences between the short and the long effect delay in the response-effect incompatible, t = 11.05, p < .001, |2β| = 249 ms, as compared to the response-effect compatible condition, t = 9.30, p < .001, |2β| = 192 ms.
|EXPERIMENTS||EXPERIMENT 1||EXPERIMENT 2|
|Intercept||432.91||407.24 – 458.57||12.39||34.94||<0.001||428.89||397.26–460.52||16.13||26.59||<0.001|
|Response-effect compatibility||–18.91||–40.28 – 2.46||10.32||–1.83||0.080||–19.58||–41.21–2.05||11.03||–1.78||0.076|
|effect delay||–110.20||–129.06 – 91.33||9.10||–12.11||<0.001||–84.18||–106.52–61.83||11.40||–7.39||<0.001|
|Response-effect compatibility × effect delay||14.18||2.77 – 25.60||5.82||2.44||0.015||10.88||2.50–18.59||4.10||2.57||0.010|
|Marginal R2/Conditional R2||0.126/0.130||0.111/0.171|
In Experiment 1, we aimed to replicate the SEC effect in anticipatory saccade frequencies (e.g., Gouret & Pfeuffer, 2021; Pfeuffer et al., 2016, 2022) as well as the influence of effect delay on anticipatory saccade latencies (Gouret & Pfeuffer; under review/preprint). Participants had to respond to a forced choice, response repeat/switch target with a left/right key press. Correct responses were predictably followed by an effect at a spatially response-effect compatible or response-effect incompatible location. One response caused a visual effect after a short effect delay of 200 ms, whereas the other response caused a visual effect after a long effect delay of 800 ms. Crucially, effects were irrelevant to participants’ task.
SEC scores were just above chance level in the response-effect compatible condition (i.e., participants looked towards future effects above chance), but did not differ from chance in the response-effect incompatible condition. This pattern is in stark contrast to the results of prior in-laboratory studies (e.g., Gouret & Pfeuffer, 2021; Pfeuffer et al., 2016, 2022). Although SEC effects were previously observed to be smaller in the response-effect incompatible than the response-effect compatible condition (e.g., Gouret & Pfeuffer, 2021; Pfeuffer et al., 2022), SEC effects in prior in-laboratory studies were, on average, substantially above chance (>70%). Moreover, it is unlikely that the lack of an SEC effect in the response-effect incompatible condition and the very subtle SEC effect in the response-effect compatible condition can be attributed to the less precise eye tracking methodology. The webcam-based eye tracking might have prevented us from registering saccades as precisely, but this should not have selectively favoured the detection of effect-incongruent saccades (i.e., saccades away from the future effect).
Thus, we concluded that aspects of the online setting itself might have had an effect on participants. A possible explanation for the difference in result patterns regarding the SEC effect online as compared to in-laboratory was found in participants’ answers to our general post-experiment questions. When asked about the usage of any strategy to complete the task, participants reported that they tried their best to focus their eyes on the middle of the screen where the target would appear and not look anywhere else (e.g., on the irrelevant effects). This aligns with the observed data patterns. Participants, overall, did not perform many eye movements away from the target area. As such, they might have suppressed anticipatory saccades towards the future effect. This strategy might have been the result of the instructions given at the beginning of the experiment. Both in-laboratory and online, participants were asked to always respond as fast and accurately as possible. However, although errors during the testing phase were not an issue either in-laboratory or online, committing too many errors during a preceding training phase was a reason for premature exclusion and, correspondingly, reduced compensation in the online experiment in contrast to the prior in-laboratory experiments to ensure compliance with instructions. This might have led to a strategy in online participants which optimized on-target attention at the cost of attending to the (irrelevant) effects.
Nevertheless, although we only found a subtle SEC effect in the response-effect compatible condition, strong saccade latency effects were replicated (Gouret & Pfeuffer, under review/preprint). That is, participants performed their first effect-congruent saccade substantially later when the effect delay was long rather than short. Interestingly, the magnitude of this effect was larger than the one reported in Gouret & Pfeuffer (under review/preprint). This first confirms that the influence of effect delay on saccade latency could be replicated online even when using participants’ individual webcams. That such a strong effect was observed even though few effect-congruent saccades were performed further attests to the reliability of the saccade latency effect itself which does not seem to be crucially dependent on the exact experimental setting or the number of saccades included in the analysis. Moreover, it indicates that even when participants try to prevent themselves from performing anticipatory saccades towards their actions’ future effects, saccade latency is a reliable indicator of effect anticipation and proactive effect monitoring – much more so than the frequency of effect-congruent saccades.
The aim of Experiment 1 was to replicate two in-laboratory findings online: The SEC effect in saccade frequencies and the influence of effect delay on saccade latency. Although substantial differences in saccade latencies were found according to effect delay, saccade frequencies were barely affected by the future effect position and only differed from chance in the response-effect compatible condition. Based on participants’ reports, we concluded that they perceived the effects as irrelevant distractions due to the online instructions. Thus, we conducted a second experiment in which we made effects relevant to assess whether stable SEC effects comparable to in-laboratory settings could be observed online under these conditions.
Experiment 2 was the same as Experiment 1 except that effects were rendered relevant to participants’ task. That is, participants now saw a number within the coloured effect circle and had to respond to it in case it was a 3.
Sample size considerations were the same as in Experiment 1. Thus, a sample of ten participants should have sufficed to replicate the influence of effect delay on saccade latencies. However, to reach a sample size that allows for comparisons with our prior studies as well as Experiment 1, we aimed for a sample size of roughly 24 participants.
Participant inclusion criteria were the same as in Experiment 1. Like in Experiment 1, 24 new participants with full datasets were recruited via Prolific, but one participant indicated that their data could not be used in a corresponding post-experiment question. Thus, 23 participants were included in the final sample (14 males, 9 females, mean age = 33.1, SD = 10.1, 4 left-handed, all of them working). Nineteen additional participants started the experiment but failed the technical requirements. They were compensated for the time they spent in the experiment. No participants who completed the experiment were excluded due to high error rates. All participants were naïve to the purpose of the experiment, provided informed consent at the beginning of the online experiment, and received 7.50 GBP as compensation for their participation. The study was conducted in agreement with the Declaration of Helsinki and the guidelines set by the local ethics committee.
Stimuli and apparatus were the same as in Experiment 1.
The design was the same as in Experiment 1 with one exception. Whenever participants produced an effect, a random number (1–9, 1°, allocated to trials equally frequently across the entire experiment) was presented in the centre of the effect, that is, a feature of the effect became response relevant. In case this number was a three, participants were to respond by pressing the space bar. Effects were still presented for 500ms, but participants could respond for 1500ms (i.e., 1000ms into the following ITI). Participants did not receive trial-wise feedback on their responses/non-responses to the effect. At the end of a block, however, they were informed about the percentage of trials with an effect they had correctly responded/not responded on.
Trial exclusions and analyses were the same as in Experiment 1 (premature responses: <0.1%, response omissions: <0.1%). Participants correctly responded to the effect number 95.0% of trials with a mean RT of 690 ms (SD = 166 ms). We followed the same analysis strategy as in Experiment 1 and reduced the random effects structure accordingly in case a model did not converge without a singular fit or negative Hessian eigenvalues.
The mean RT was 464ms (SD = 160ms). The LMM fitting manual RTs included participant intercepts and by-participant random slopes for response-effect compatibility and effect delay as random effect (see Figure 2C). response-effect compatibility was significant, t (23) = 4.21, p < .001, |2β| = 12.60 ms. Participants responded faster in the response-effect compatible condition than in the response-effect incompatible condition. Effect delay and the interaction of response-effect compatibility and effect delay did not reach significance, |ts| ≤ 0.91, ps ≥ .364 (see Table 1).
The mean error rate was 4.6% (SD = 3.0%). The GLMM fitting errors included participant intercepts and by-participant random slopes for response-effect compatibility and effect delay as random effect (see Figure 2D). As in Experiment 1, no significant effects were observed, |zs| ≤ 0.76, ps ≥ .450 (see Table 2).
Saccade selection criterion were the same as in Experiment 1. Only saccades occurring within the anticipatory interval (target offset to effect onset) fulfilling the inclusion criteria were considered. A total of 4,260 saccades were used for the saccade-effect congruency analysis and 3,780 saccades were considered for the saccade latency analysis. Again, to infer tracking precision, we additionally assessed gaze sample distributions during target presentation. The mean dispersion of gaze samples in X direction during target presentation was 0.7° and the mean dispersion of gaze samples in Y direction was 1.5°. That is, gaze sample dispersion in a phase of the experiment in which participants should fixate one spot, the target location, was lower than the 2° saccade criterion we set to detect saccades.
We first conducted two one-sample t-tests to assess participants’ overall SEC scores. These one-sample t-tests showed that participants mean SEC were significantly greater than 50% in the response-effect compatible condition, t(22) = 11.4, p < .001, d = 2.38 (M = 74.8%, SD = 10.4%), as well as in the response-effect incompatible condition, t(22) = 9.01, p < .001, d = 1.89 (M = 66.9%, SD = 8.9%).
The GLMM fitting saccade-effect congruency (0 = effect-incongruent, 1 = effect-congruent) included participant intercepts and by-participant random slopes for response-effect compatibility and effect delay (see Figure 3C). Response-effect compatibility reached significance, z = 7.25, p < .001, OR = 2.96. Participants performed more effect-congruent saccades when the future effect’s location was spatially response-effect compatible rather than incompatible with their response. Moreover, the interaction of response-effect compatibility and effect delay was significant, z = 2.66, p = .008, OR = 2.20 (see Table 3 for detailed results). Participants did not show larger SEC differences between the short and long effect delay in the response-effect compatible, z = 1.52, p = .128, OR = 0.20, or in the response-effect incompatible condition, z = 1.29, p = .198, OR = 0.17. The interaction emerged due to opposing patterns. Effect delay did not yield a significant influence, z = 0.17, p = .864, OR = 2.02.
Again, only the first effect-congruent saccade performed per trial was considered. The LMM fitting participants’ first effect-congruent saccades’ latency included participant intercepts and by-participant random slopes for response-effect compatibility and effect delay as random effect (see Figure 3D). The response-effect compatibility effect was not significant, t = 1.78, p = .076, |2β| = 39 ms. Effect delay reached significance, t = 7.39, p < .001, |2β| = 168 ms. That is, participants’ first effect-congruent saccade was performed later for the long as compared to the short effect delay. Effect-delay and response-effect compatibility significantly interacted, t = 2.57, p = .010, |2β| = 22 ms (see Table 4). Participants showed larger latency differences between the short and long effect delay in the response-effect incompatible, t = 7.78, p < .001, |2β| = 192 ms, than in the response-effect compatible condition, t = 6.19, p < .001, |2β| = 149 ms.
In Experiment 2, we modified the design of Experiment 1 by changing the relevance of the visual effect following participants’ responses. In Experiment 2, effects were relevant as compared to Experiment 1 in which they were irrelevant to participants’ task. That is, in Experiment 2, participants’ actions produced a coloured circle with a number at its centre and participants had to pay attention to the number and respond to it if it was a 3.
In contrast to Experiment 1, participants looked more often towards the future effect than expected by chance in both the response-effect compatible and response-effect incompatible condition, replicating both the result patterns and magnitudes (at least on response-effect compatible trials) found in prior in-laboratory studies (e.g., Gouret & Pfeuffer, 2021; Pfeuffer et al., 2016, 2022). This result corroborates that effect relevance supports higher anticipatory saccade frequencies and allows for observing stable SEC effects online.
In line with Experiment 1 and a prior in-laboratory study (Gouret & Pfeuffer, under review/preprint), a saccade latency effect was again observed in Experiment 2. Participants performed their first effect-congruent saccade later when the effect delay was long rather than short. In size, this saccade latency effect appeared to be comparable to Experiment 1. This further emphasizes the reliability of anticipatory saccade latencies as an excellent indicator of effect anticipation and proactive effect monitoring under various experimental conditions.
Across two online eye tracking experiments, we addressed the question whether anticipatory saccades – specifically frequency and latency effects observed in prior in-laboratory studies (Gouret & Pfeuffer, 2021, under review/preprint; Pfeuffer et al., 2016, 2022) – could be replicated online using participants’ individual webcams. In both experiments, participants’ actions, after a response-specific short/long effect-delay, led to the appearance of a visual effect predictably located at a position spatially compatible/incompatible to their response. Effects were irrelevant to participants’ task in Experiment 1, whereas they were relevant in Experiment 2.
Anticipatory saccade frequency effects were replicated in Experiment 2, but only very subtly present and only in the response-effect compatible condition in Experiment 1. In in-laboratory studies, the SEC effect, was consistently found to be strong (Pfeuffer et al., 2016, 2022; Gouret & Pfeuffer, 2021; under review/preprint; comparable to its size in Experiment 2). In line with our reasoning for manipulating effect relevance, this suggests that, depending on task demands, participants can suppress anticipatory saccades (e.g., when target attention is deemed as more essential). From a theoretical perspective, this is a very interesting finding as it questions to what degree proactive effect monitoring can be considered an automatic process (see e.g. Moors & De Houwer, 2006, for a discussion of different theoretical positions on automaticity). Comparing the findings of Experiment 1 to prior in-laboratory studies, it appears that participants only perform a substantial number of anticipatory saccades when attentional task demands are relaxed. This also aligns with the observation that anticipatory saccade frequencies substantially differ between individuals. Furthermore, this means that proactive effect monitoring (or at least its expression in anticipatory saccades) might rely on a cognitive resource that is also required for task-related processing and/or attention.
Importantly, effect relevance appeared to overrule participants’ strategy to suppress anticipatory saccades. When effects were relevant, participants again performed anticipatory saccades towards the location of their actions’ future effects well above chance level. Furthermore, our findings suggest that effect relevance mainly modulates the occurrence/frequency but not latency of anticipatory saccades, as effect delay comparably affected saccade latencies in both experiments. This extends prior in-laboratory studies (Pfeuffer et al., 2016, 2022; Gouret & Pfeuffer, 2021; under review/preprint) and also allows for the methodological recommendation to use relevant effects (or effect features) when it is essential to generate a large number of anticipatory saccades and/or when participants might try to suppress anticipatory saccades, as it might, for instance, be the case in online settings.
Moreover, even when anticipatory saccades were infrequent and SEC effects were not observable in saccade frequencies, saccade latencies nonetheless reliably reflected that participants anticipatorily monitored the timing of their actions’ future effects. That is, we observed longer saccade latencies for long than for short effect delays irrespective of SEC effects (or effect relevance). Methodologically, this highlights anticipatory saccade latencies as the more sensitive and reliable measure of proactive effect monitoring that can be observed even under suboptimal eye tracking conditions (e.g., online participants’ using individual webcams).
From a theoretical perspective, these observations clearly differentiate between saccade frequency and saccade latency effects. Saccade latency effects can be found irrespective of SEC effects and, in contrast to SEC effects, they did not seem to be affected by effect relevance. This suggests that these effects could emerge due to different subprocesses of proactive effect monitoring. This finding is also in line with prior studies on effect-generating actions which suggest that spatial and temporal features of an action’s future effect might independently be associated with the action and/or be anticipated independently (e.g., Dignath & Janczyk, 2017; Pfister et al., 2017). Building on the present findings and assessing whether there are multiple subprocesses of proactive effect monitoring will further increase our understanding of proactive effect monitoring and goal-directed action control.
Finally, our findings demonstrate that anticipatory saccades are a reliable measure of proactive effect monitoring even under suboptimal eye tracking conditions, for instance, during webcam-based online eye tracking. Our findings illustrated that an average webcam and computer, resulting in sampling rates below 30 Hz, are sufficient to assess anticipatory saccades. First, this replication under adversarial conditions speaks to the reliability of anticipatory saccade latency effects and (for relevant effects) anticipatory saccade frequency effects. Second, it broadens the range of settings in which anticipatory saccades can be assessed. On the one hand, they can be used in basic research on human action control even when one does not have an eye tracker with high temporo-spatial resolution at hand. On the other hand, that anticipatory saccades (i.e., proactive effect monitoring) can also be assessed online with participants’ webcams at below 30 Hz, opens up the chance of using anticipatory saccades as a measure in applied context, especially human-technology interaction settings, in the future.
The data as well as experiment files and syntaxes are publicly available on OSF: https://doi.org/10.17605/OSF.IO/AVFBX.
1Note that, given our contrast coding, β represents the difference between one condition and the grand mean. When comparing two conditions as in the reported analyses, 2β thus describes the size of the difference between conditions, that is, the effect size.
This study was performed in line with the principles of the Declaration of Helsinki and the University of Freiburg. All participants gave their informed consent before participation.
We thank Jonas Plate for his help in setting up the online eye tracking experiments.
This research was supported by a grant of the Deutsche Forschungsgemeinschaft (DFG) awarded to Christina Pfeuffer (PF 893/1–1, Christina U. Pfeuffer), a grant of the Baden-Württemberg Stiftung awarded to Christina Pfeuffer who is indebted to the Baden-Württemberg Stiftung for the financial support of this research project by the Eliteprogramme for Postdocs as well as by the German Research Foundation (DFG) within the “Open Access Publication Funding” programme.
The authors have no competing interests to declare.
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting Linear Mixed-Effects Models Usinglme4. Journal of Statistical Software, 67(1). DOI: https://doi.org/10.18637/jss.v067.i01
Del Popolo Cristaldi, F., Granziol, U., Bariletti, I., Borella, E., & Mento, G. (2022). Doing experimental psychological research in pandemic time: how alerting differently impacts online vs. lab settin. Europe PMC. DOI: https://doi.org/10.31234/osf.io/cn3dj
Dignath, D., & Janczyk, M. (2017). Anticipation of delayed action-effects: learning when an effect occurs, without knowing what this effect will be. Psychological Research, 81(5), 1072–1083. DOI: https://doi.org/10.1007/s00426-016-0797-7
Dignath, D., Pfister, R., Eder, A. B., Kiesel, A., & Kunde, W. (2014). Representing the hyphen in action–effect associations: Automatic acquisition and bidirectional retrieval of action–effect intervals. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(6), 1701–1712. DOI: https://doi.org/10.1037/xlm0000022
Dutzi, I. B., & Hommel, B. (2009). The microgenesis of action-effect binding. Psychological Research, 73(3), 425–435. DOI: https://doi.org/10.1007/s00426-008-0161-7
Elsner, B., & Hommel, B. (2001). Effect anticipation and action control. Journal of Experimental Psychology: Human Perception and Performance, 27(1), 229–240. DOI: https://doi.org/10.1037/0096-1522.214.171.124
Elsner, B., & Hommel, B. (2004). Contiguity and contingency in action-effect learning. Psychological Research, 68(2–3), 138–154. DOI: https://doi.org/10.1007/s00426-003-0151-8
Elsner, B., Hommel, B., Mentschel, C., Drzezga, A., Prinz, W., Conrad, B., & Siebner, H. (2002). Linking Actions and Their Perceivable Consequences in the Human Brain. NeuroImage, 17(1), 364–372. DOI: https://doi.org/10.1006/nimg.2002.1162
Gouret, F., & Pfeuffer, C. U. (2021). Learning to expect and monitor the future: How fast do anticipatory saccades toward future action consequences emerge? Journal of Experimental Psychology: Human Perception and Performance, 47(7), 992–1008. DOI: https://doi.org/10.1037/xhp0000924
Gouret, F., & Pfeuffer, C. U. (under review/preprint). One-shot learning of proactive effect monitoring: Temporally-distinct attention allocation towards future action outcomes. Preprint available online on ResearchGate.
Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should We Trust Web-Based Studies? A Comparative Analysis of Six Preconceptions About Internet Questionnaires. American Psychologist, 59(2), 93–104. DOI: https://doi.org/10.1037/0003-066X.59.2.93
Halekoh, U., & Højsgaard, S. (2014). A kenward-roger approximation and parametric bootstrap methods for tests in linear mixed models–the R package pbkrtest. Journal of Statistical Software, 59(9), 1–30. DOI: https://doi.org/10.18637/jss.v059.i09
Hoffmann, J. (2003). Anticipatory Behavioral Control. Anticipatory Behavior in Adaptive Learning Systems, 44–65. DOI: https://doi.org/10.1007/978-3-540-45002-3_4
Hommel, B. (2009). Action control according to TEC (theory of event coding). Psychological Research, 73(4), 512–526. DOI: https://doi.org/10.1007/s00426-009-0234-2
Hommel, B., Müsseler, J., Aschersleben, G., & Prinz, W. (2001). The Theory of Event Coding (TEC): A framework for perception and action planning. Behavioral And Brain Sciences, 24(5), 849–878. DOI: https://doi.org/10.1017/S0140525X01000103
Innes, R. J., Evans, N. J., Howard, Z. L., Eidels, A., & Brown, S. D. (2020). A Broader Application of the Detection Response Task to Cognitive Tasks and Online Environments. Human Factors: The Journal of the Human Factors and Ergonomics Society, 63(5), 896–909. DOI: https://doi.org/10.1177/0018720820936800
James, W. (1981). The principles of psychology. Cambridge, MA: Harvard University Press (Original work published 1890).
Johansen, S. A., San Agustin, J., Skovsgaard, H., Hansen, J. P., & Tall, M. (2011). Low cost vs. high-end eye tracking for usability testing. Proceedings of the 2011 Annual Conference Extended Abstracts on Human Factors in Computing Systems – CHI EA ’11. DOI: https://doi.org/10.1145/1979742.1979744
Kenward, B. (2010). 10-month-olds visually anticipate an outcome contingent on their own action. Infancy, 15(4), 337–361. DOI: https://doi.org/10.1111/j.1532-7078.2009.00018.x
Kenward, M. G., & Roger, J. H. (1997). Small Sample Inference for Fixed effects from Restricted Maximum Likelihood. Biometrics, 53(3), 983. DOI: https://doi.org/10.2307/2533558
Koch, I., & Kunde, W. (2002). Verbal response-effect compatibility. Memory & Cognition, 30(8), 1297–1303. DOI: https://doi.org/10.3758/BF03213411
Kumle, L., Võ, M. L. H., & Draschkow, D. (2021). Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R. Behavior Research Methods, 53(6), 2528–2543. DOI: https://doi.org/10.3758/s13428-021-01546-0
Kunde, W. (2001). Response-effect compatibility in manual choice reaction tasks. Journal of Experimental Psychology: Human Perception and Performance, 27(2), 387–394. DOI: https://doi.org/10.1037/0096-15126.96.36.1997
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017). lmerTest Package: Tests in Linear Mixed effects Models. Journal of Statistical Software, 82(13). DOI: https://doi.org/10.18637/jss.v082.i13
Lenth, R., Singmann, H., Love, J., Buerkner, P., & Herve, M. (2018). Emmeans: Estimated Marginal Means, AKA Least-squares Means. R Package Version 1.1. 3; 2018. https://CRAN.R-project.org/package=emmeans
Moors, A., & De Houwer, J. (2006). Automaticity: a theoretical and conceptual analysis. Psychological Bulletin, 132(2), 297–326. DOI: https://doi.org/10.1037/0033-2909.132.2.297
Nakagawa, S., Johnson, P. C. D., & Schielzeth, H. (2017). The coefficient of determination R2 and intra-class correlation coefficient from generalized linear mixed-effects models revisited and expanded. Journal of The Royal Society Interface, 14(134), 20170213. DOI: https://doi.org/10.1098/rsif.2017.0213
Nakagawa, S., & Schielzeth, H. (2013). A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2), 133–142. DOI: https://doi.org/10.1111/j.2041-210x.2012.00261.x
Papoutsaki, A., Laskey, J., & Huang, J. (2017). SearchGazer: Webcam Eye Tracking for Remote Studies of Web Search. Proceedings of the ACM SIGIR Conference on Human Information Interaction Retrieval (CHIIR). DOI: https://doi.org/10.1145/3020165.3020170
Pfeuffer, C. U., Kiesel, A., & Huestegge, L. (2016). A look into the future: Spontaneous anticipatory saccades reflect processes of anticipatory action control. Journal of Experimental Psychology: General, 145(11), 1530–1547. DOI: https://doi.org/10.1037/xge0000224
Pfeuffer, C. U., Kiesel, A., & Huestegge, L. (2022). Similar proactive effect monitoring in free and forced choice action modes. Psychological Research. Advance online publication. DOI: https://doi.org/10.1007/s00426-022-01644-4
Pfister, R., Janczyk, M., Wirth, R., Dignath, D., & Kunde, W. (2014). Thinking with portals: Revisiting kinematic cues to intention. Cognition, 133(2), 464–473. DOI: https://doi.org/10.1016/j.cognition.2014.07.012
Pfister, R., Kiesel, A., & Melcher, T. (2010). Adaptive control of ideomotor effect anticipations. Acta Psychologica, 135(3), 316–322. DOI: https://doi.org/10.1016/j.actpsy.2010.08.006
Pfister, R., Pfeuffer, C. U., & Kunde, W. (2014). Perceiving by proxy: Effect-based action control with unperceivable effects. Cognition, 132(3), 251–261. DOI: https://doi.org/10.1016/j.cognition.2014.04.012
Pfister, R., Weller, L., Dignath, D., & Kunde, W. (2017). What or when? The impact of anticipated social action effects is driven by action-effect compatibility, not delay. Attention, Perception & Psychophysics. DOI: https://doi.org/10.3758/s13414-017-1371-0
Semmelmann, K., & Weigelt, S. (2016). Online psychophysics: reaction time effects in cognitive experiments. Behavior Research Methods, 49(4), 1241–1260. DOI: https://doi.org/10.3758/s13428-016-0783-4
Semmelmann, K., & Weigelt, S. (2018). Online webcam-based eye tracking in cognitive science: A first look. Behavior Research Methods, 50(2), 451–465. DOI: https://doi.org/10.3758/s13428-017-0913-7
Shin, Y. K., Proctor, R. W., & Capaldi, E. J. (2010). A review of contemporary ideomotor theory. Psychological bulletin, 136(6), 943-974. DOI: https://doi.org/10.1037/a0020541
Singmann, H., Bolker, B., Westfall, J., & Aust, F. (2015). afex: Analysis of factorial experiments. R package version 0.13–145.
Sun, D., Custers, R., Marien, H., & Aarts, H. (2020). Ideomotor Action: Evidence for Automaticity in Learning, but Not Execution. Frontiers in Psychology, 11. DOI: https://doi.org/10.3389/fpsyg.2020.00185
Szumilas, M. (2010). Explaining odds ratios. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 19(3), 227–229. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/20842279
Wickham, H. (2016). ggplot2: Elegant Graphics for Data Analysis. Springer. DOI: https://doi.org/10.1007/978-3-319-24277-4
Wirth, R., Pfister, R., Janczyk, M., & Kunde, W. (2015). Through the portal: effect anticipation in the central bottleneck. Acta Psychologica, 160, 141–151. DOI: https://doi.org/10.1016/j.actpsy.2015.07.007