The Forward Testing Effect is Reliable and Independent of Learners’ Working Memory Capacity

Bernhard Pastötter; Christian Frings

doi:10.5334/joc.82

Introduction

Tests are commonly used to evaluate learning. However, tests can also be used to enhance learning and retention. This has been demonstrated in both basic research in the laboratory (for reviews, see Karpicke, 2017; Roediger & Butler, 2011; Rowland, 2014) and applied research in educational contexts (for reviews, see Adesope, Trevisan, & Sundararajan, 2017; Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013; Moreira, Pinto & Starling, 2019). Thereby, testing can have both direct and indirect effects on long-term learning and retention (Roediger & Karpicke, 2006a). A prominent direct benefit of testing is the (backward) testing effect, which refers to the finding that when participants (repeatedly) retrieve versus restudy previously studied item material, the retrieved items are better recalled on a delayed test than the restudied items (e.g., Hogan & Kintsch, 1971; Roediger & Karpicke, 2006b). The backward testing effect is a robust phenomenon that generalizes across a wide range of materials and experimental settings (see Karpicke, 2017). Different mechanisms have been suggested to contribute to the backward testing effect. For instance, retrieval practice, more than restudy, may promote semantic elaboration (Carpenter, 2009) and/or may enhance contextual processing of the information (Karpicke, Lehman, & Aue, 2014), thus enhancing learning and memory accessibility of the practiced information.

In addition, there are indirect benefits of testing. Actually, it is these indirect benefits of testing that students typically aim for when practicing retrieval on their own, not the direct benefits of testing (Karpicke, Butler, & Roediger, 2009). For instance, retrieval practice of previously studied material potentiates relearning of the practiced information, a finding that has been termed test-potentiated learning (Arnold & McDermott, 2013; Izawa, 1971; see also Tempel & Frings, 2018). The typical experiment consists of a study phase, a restudy phase, and a final test phase. Critically, between study and restudy phases, it is manipulated whether the material is tested or not. The typical finding is that interpolated testing promotes relearning of the studied material and enhances memory performance in the final test. On the other hand, retrieval practice also potentiates subsequent learning of other information, a finding that has been referred to as forward testing effect (Pastötter & Bäuml, 2014) or test-potentiated new learning (Chan, Meissner, & Davis, 2018). The forward testing effect is particularly striking because it is on the learning of new information that is not necessarily related to the previously studied (and tested) item material (see Yang, Chew, Sun, & Shanks, 2019). The forward testing effect can be studied in interim-testing tasks: Participants study several (e.g., three) lists of items in anticipation of a final cumulative recall test. Between the study of lists, participants are either tested on the most recent list (i.e., list 1 or list 2) in a recall test, or restudy the items of the most recent list. After study of all lists, participants are tested on the last list (i.e., list 3), which is the critical list. The typical finding is that interim testing of lists 1 and 2 enhances recall of list 3 and reduces the number of prior-list intrusions in the list 3 recall test (e.g., Bäuml & Kliegl, 2013; Szpunar, McDermott, & Roediger, 2008; Weinstein, McDermott, & Szpunar, 2011; Yang et al., 2019). The forward testing effect has been observed with different item materials (e.g., words, pictures, texts, videos), for different numbers of to-be-studied item lists (e.g., two lists, three lists, five lists) and for different types of recall tasks used in the interim tests (e.g., cued recall, free recall; for reviews, see Pastötter & Bäuml, 2014; Yang, Potts, & Shanks, 2018; for a meta-analysis on the forward testing effect, see Chan et al., 2018).

Both encoding and retrieval accounts have been put forth to account for the forward testing effect (see Yang et al., 2018, for a recent review). Retrieval accounts assume that recall testing between the study of lists promotes contextual list segregation, which may enhance list differentiation and reduce interference between lists at test (Bäuml & Kliegl, 2013; Szpunar et al., 2008). In contrast, encoding accounts assume that testing of previously studied lists improves encoding of subsequently studied item material. Specifically, it has been suggested that interim testing may reduce memory load (i.e., interference during encoding) and enhance attentional encoding, which makes the encoding of later lists as effective as the encoding of earlier lists (Pastötter et al., 2018; Pastötter, Schicker, Niedernhuber, & Bäuml, 2011), or a change in participants’ encoding strategy, which enhances elaborative encoding for the later lists compared to the earlier lists (Chan, Manley, Davis, & Szpunar, 2018; Wissman, Rawson, & Pyc, 2011; see also Garcia-Marques, Nunes, Marques, Carneiro, & Weinstein, 2015). In addition, motivational factors have been suggested that may mediate the encoding and/or retrieval benefits of testing. Accordingly, retrieval failures in tests may induce dissatisfaction in learners and thus motivate more effort toward subsequent encoding and/or retrieval of new information (Cho, Neely, Crocco, & Vitrano, 2017). Furthermore, interim testing may induce in learners a greater expectancy of additional tests, which may motivate more effort toward attentional encoding of new information (Weinstein, Gilmore, Szpunar, & McDermott, 2014). Thus, multiple encoding and retrieval factors and different mediators may contribute to the forward testing effect. Notably, the same factors may also contribute to test-potentiated learning, as has been suggested in a recent study by Cho et al. (2017). These authors demonstrated comparable indirect effects of interpolated testing on the relearning and final recall of previously studied, old items (test-potentiated learning) and the learning and recall of previously not studied, new items (forward testing effect).

Regarding individual-differences research on testing effects, a considerable amount of studies have examined possible influences of learners’ age and working memory capacity (WMC) on testing effects. This research has established both direct and indirect testing effects in children, young people, and older adults. For instance, children in (late) primary school benefit from the backward testing effect (Karpicke, Blunt, & Smith, 2016), the forward testing effect (Aslan & Bäuml, 2016), and test-potentiated learning (Lipowski, Pyc, Dunlosky, & Rawson, 2014), and even preschool children may benefit from some forms of retrieval practice (Kliegl, Abel, & Bäuml, 2018; see Fazio & Marsh, in press, for a review). In addition, several studies have established both backward (Meyer & Logan, 2013) and forward testing effects (Pastötter & Bäuml, 2019) and test-potentiated learning (Coane, 2013) in older adults, with the size of the effects comparable to younger adults. Furthermore, reliable testing effects were observed in memory-impaired patient groups, including patients with multiple sclerosis, traumatic brain injury, and Alzheimer’s disease (Balota, Duchek, Sergent-Marshall, & Roediger, 2006; Pastötter, Weber, & Bäuml, 2013; Sumowski, Chiaravalloti, & DeLuca, 2010; Sumowski, Wood et al., 2010; but see Pastötter, Eberle, Aue, & Bäuml, 2017).

Regarding the relationship between WMC and the benefits of retrieval practice, several studies have examined whether complex working memory tasks, such as the operation or symmetry span tasks, relate to the backward testing effect or test-potentiated learning. WMC is an interesting individual-differences variable in cognitive psychology because it accounts for a substantial portion of variance in a person’s general fluid intelligence and executive function (see Conway, Kane, & Engle, 2003; Foster et al., 2015; McCabe, Roediger, McDaniel, Balota, & Hambrick, 2010). The majority of these studies found no (direct) relationship between WMC and the backward testing effect effect (Agarwal, Finley, Rose, & Roediger, 2017; Aslan & Bäuml, 2011) or test-potentiated learning (Bertilsson, Wiklund-Hörnqvist, Stenlund, & Jonsson, 2017; Brewer & Unsworth, 2012; Minear, Coane, Boland, Cooney, & Albat, 2018; Tsu & Pu, 2012; Wiklund-Hörnqvist, Jonsson, & Nyberg, 2014; but see Agarwal et al., 2017). Notably, the study by Tse and Pu (2012) showed that a significant proportion of variance in test-potentiated learning could be predicted by an interaction between participants’ WMC and test anxiety scores. Thus, more indirect relationships between WMC and the benefits of retrieval practice may exist. To the best of our knowledge, no study has yet examined the relationship between the forward testing effect and participants’ WMC.

Overview

In two experiments we investigated individual differences in the forward testing effect. In both experiments, participants went through both a testing and a restudy condition. In each condition, participants studied three lists of words, which they were asked to remember for a final recall test. In the testing condition, they were tested immediately on lists 1 and 2 after studying each single list, whereas in the restudy condition, they restudied lists 1 and 2 after initial study. In both conditions, participants were tested immediately on list 3 and finally on lists 1 and 2. The critical variables were correct recall and the number of prior-list intrusions in the immediate list 3 recall test. Individual forward-testing-effect recall and intrusion scores (shortly referred to as FTE recall and intrusion scores in the following) were determined by calculating individual differences between conditions. Experiment 1 examined the test-retest reliability of the forward testing effect, i.e., the correlation of FTE recall and intrusion scores between two experimental sessions. Experiment 2 examined the relationship between the forward testing effect and participants’ WMC, measured with complex WM span tasks. Because the reliability of a measure limits the correlation that can be observed with another measure (Nunally, 1970; Spearman, 1904), significant test-retest reliability of FTE scores in Experiment 1 was considered a necessary precondition for analyzing the relationship of these scores to participants’ WMC in Experiment 2. Based on the finding that test-potentiated learning is not related to learners’ WMC (Brewer & Unsworth, 2012; Wiklund-Hörnqvist et al., 2014) and the view that the same factors contribute to test-potentiated learning and the forward effect of testing (Cho et al., 2017), we hypothesized to find no significant correlation between the forward testing effect and participants’ WMC. To foreshadow the results, the data of Experiment 1 suggested sufficient reliability of the forward testing effect and thereby enabled us to analyze the potential relationship of the effect to individual WMC. In accordance with our hypothesis we did not find evidence for a correlation. Note that as we actually predicted a null correlation, Bayesian statistics are reported in addition to frequentist statistics.

Experiment 1

The goal of Experiment 1 was to examine test-retest reliability of the forward testing effect. Participants took part in two sessions, which both consisted of a testing and a restudy condition. The two sessions were one week apart. Different items were used in the two sessions. FTE recall and intrusion scores were calculated for both sessions and test-retest reliabilities were estimated using Intraclass Correlation Coefficients (ICC; see Hedge, Powell, & Sumner, 2018; Parsons, Kruijt, & Fox, 2018).

Method

Participants

Required sample size for the test-retest reliability analysis in Experiment 1 was calculated based on methods suggested by Walter, Eliasziw, and Donner (1998). Given α = 0.05 and a desired power of 1 – β = 0.95 to observe a fair level of reliability with ICC of 0.3 (or larger), a required sample size of 57 subjects was calculated. Note that ICCs of 0.8, 0.6, and 0.4 are typically interpreted as excellent, good/substantial, and moderate levels of reliability, respectively (Fleiss, 1981; Landis & Koch, 1977).

Sixty-four students from the University of Trier participated in Experiment 1 (mean age: 22.4 years, SD = 2.5 years; 24 females, 40 males). All participants successfully completed both sessions. The study was carried out in accordance with the recommendations of the Declaration of Helsinki and approved by the local ethical review committee at the University of Trier (reference number: 50/2017). All participants gave written informed consent.

Material

The material was taken from a study by Pastötter, Kliegl, and Bäuml (2012; Exp. 2). It consisted of 144 unrelated German nouns of medium frequency and word length of 4 to 8 letters, which were drawn from CELEX database (Duyck, Desmet, Verbeke, & Brysbaert, 2004). The 144 items were randomly assigned to twelve 12-item lists. Six lists were used in session 1, the other six lists in session 2. In each session, three lists were assigned to the testing condition, the other three lists were assigned to the restudy condition. Thus, different items were used in the two sessions.

Procedure

The two sessions were one week apart. Participants were retested in session 2 at about the same time as they were tested in session 1, with maximum deviation of two hours between sessions. In both sessions, participants took part in both the testing and the restudy condition, with order of conditions counterbalanced across participants. In both conditions, participants studied three 12-item lists (see Figure 1). Items of each list were visually presented in random order in the middle of a computer screen with a presentation rate of 3.75 sec (3 sec item presentation, 0.75 sec blank screen). Subsequent to the presentation of each list (45 sec), all participants solved simple mathematics (e.g., 4 * 8) for 30 sec as a distractor. Experimental conditions differed in inter-list activity that followed the distractors after lists 1 and 2. In the testing condition, after study of list 1 and the distractor, participants were given 45 sec to recall in any order they wished as many items as they could remember from list 1; next, after study of list 2 and the distractor, participants were given 45 sec to recall in any order they wished as many items as they could remember from list 2. In the restudy condition, participants were instead re-presented the just-studied items of lists 1 and 2 (45 sec each) with new random item presentation order. In both conditions, the distractor after list 3 was followed by an immediate list 3 recall test, in which participants were given 45 sec to recall in any order they wished as many items they could remember from list 3. After the immediate list 3 recall test, lists 2 and 1 were tested in two final recall tests; list 2 was always tested first. Each test took 45 sec. In all recall tests, participants typed in responses on a computer keyboard.

Figure 1

Procedure of the episodic memory task. In both the testing and the restudy condition, participants studied three lists of items. Each list consisted of 12 words and was followed by a short distractor (D). List 3 was tested immediately (after the distractor) in both conditions. Lists 1 and 2 were also tested immediately in the testing condition, but were restudied in the restudy condition. After immediate recall of list 3, lists 1 and 2 were tested in final recall tests.

Data analysis

On the group level, percent correct recall in the immediate list 3 recall test was examined as a function of experimental condition (testing, restudy) and session (session 1, session 2). In addition, in the testing condition, immediate list recall was examined as a function of list (list 1, list 2, list 3) and session (session 1, session 2). Regarding final recall, percent correct list recall was examined as a function of condition (testing, restudy), list (list 1, list 2), and session (session 1, session 2). Furthermore, number of prior-list intrusions were examined in the immediate list 3 recall test; list 1 and list 2 items that were falsely recalled by participants in the immediate list 3 recall were considered as intrusions. In all analyses of variance (ANOVAs), Greenhouse-Geisser (GG) correction was applied where necessary.

Bayesian statistics were calculated using JASP 0.9.2.0 software (Wagenmakers et al., 2018). BF₀₁ is reported when the Bayesian analysis provides relatively more evidence for the null hypothesis than for the alternative hypothesis; BF₁₀ is reported when the analysis provides relatively more evidence for the alternative hypothesis than for null hypothesis.

On the item level, FTE recall and intrusion scores were calculated, separately for sessions 1 and 2. The FTE recall score was calculated by subtracting participants’ immediate list 3 recall rates in the restudy condition from their immediate list 3 recall rates in the testing condition; the FTE intrusion score was calculated by subtracting participants’ number of prior-list intrusions in the restudy condition from their number of prior-list intrusions in the testing condition. Test-retest reliability was estimated with the ICC-2k, which uses a two-way random effects model for absolute agreement (see Hedge et al., 2018; Parsons et al., 2018).

Results

Group level

The results of all recall tests are shown in Table 1. Regarding recall rates in the immediate list 3 recall test, a two-way ANOVA with the within-subjects factors of condition (testing vs. restudy) and session (session 1 vs. session 2) revealed a significant main effect of condition, F(1,63) = 36.41, MSE = 400.95, p < .001, $η p 2 = .366$ M1 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ \eta _p^2 = .366 \] \end{document} , (BF₁₀ > 100, compared to null model), a significant main effect of session, F(1,63) = 7.77, MSE = 321.90, p = .007, $η p 2 = .110$ M2 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ \eta _p^2 = .110 \] \end{document} (BF₁₀ = 2.52, compared to null model), and a significant interaction between the two factors, F(1,63) = 7.61, MSE = 228.17, p = .008, $η p 2 = .108$ M3 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ \eta _p^2 = .108 \] \end{document} (BF₁₀ = 2.30, compared to two-main-effects model). Thus, while the forward effect on list 3 recall was significant in both sessions, ps ≤ .001, the effect was smaller in session 2 than in session 1.

Table 1

Results in the Immediate and Final Recall Tests as a Function of Practice Condition and Session in Experiment 1. Means and Standard Errors of the Mean (in Parentheses).

Recall Rates						Intrusions

Session	Test	Condition	List 1	List 2	List 3	List 3

Session 1	Immediate Recall	Testing	66.93 (2.75)	68.88 (2.57)	69.01 (2.56)	0.17 (0.05)
		Restudy			48.70 (3.68)	0.36 (0.10)
		Difference			20.31 (3.29)**	–.19 (.11)
	Final Recall	Testing	41.28 (4.11)	56.64 (3.54)
		Restudy	51.69 (3.97)	55.08 (4.00)
		Difference	–10.42 (4.40)*	1.56 (3.87)

Session 2	Immediate Recall	Testing	69.79 (2.93)	68.49 (2.84)	70.05 (2.55)	0.05 (0.03)
		Restudy			60.16 (3.17)	0.19 (0.08)
		Difference			9.90 (2.97)**	–.14 (.08)
	Final Recall	Testing	38.93 (4.50)	49.61 (4.41)
		Restudy	58.98 (4.08)	61.98 (3.86)
		Difference	–20.05 (5.53)**	–12.37 (4.89)*

Pair-wise t tests (two sided, uncorrected) for differences between conditions (testing vs. restudy). * p < .05, ** p < .01.

Regarding intrusion rates in the immediate list 3 recall test, a two-way ANOVA with the within-subjects factors of condition (testing vs. restudy) and session (session 1 vs. session 2) showed a significant main effect of condition, F(1,63) = 5.56, MSE = 0.310, p = .022, $η p 2 = .081$ M4 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ \eta _p^2 = .081 \] \end{document} (BF₁₀ = 2.18, compared to null model), and a significant main effect of session, F(1,63) = 4.26, MSE = 0.331, p = .043, $η p 2 = .063$ M5 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ \eta _p^2 = .063 \] \end{document} (BF₁₀ = 1.29, compared to null model), but no significant interaction between the two factors, F(1,63) < 1 (BF₀₁ = 4.75, compared to two-main-effects model).

Regarding immediate recall of lists 1 to 3 in the testing condition, a two-way ANOVA with the within-subjects factors of list (list 1 vs. list 2 vs. list 3) and session (session 1 vs. session 2) showed no significant main effects or interaction, Fs < 1 (BF₀₁s > 7.00), indicating similar recall rates for lists 1 to 3 in the testing condition in both sessions.

With regard to final recall rates, a three-way ANOVA with the within-subjects factors of condition (testing vs. restudy), list (list 1 vs. list 2), and session (session 1 vs. session 2) was calculated. The analysis showed a significant main effect of condition, F(1,63) = 15.35, MSE = 887.75, p < .001, $η p 2 = .196$ M6 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ \eta _p^2 = .196 \] \end{document} (BF₁₀ > 100, compared to null model), a significant main effect of list, F(1,63) = 18.23, MSE = 461.35, p < .001, $η p 2 = .224$ M7 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ \eta _p^2 = .224 \] \end{document} (BF₁₀ = 28.83, compared to null model), a significant interaction between the two factors of condition and list, F(1,63) = 5.07, MSE = 609.66, p = .028, $η p 2 = .075$ M8 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ \eta _p^2 = .075 \] \end{document} (BF₁₀ = 1.23, compared to three-main-effects model), and a significant interaction between the two factors of condition and session, F(1,63) = 4.92, MSE = 902.38, p = .030, $η p 2 = .073$ M9 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ \eta _p^2 = .073 \] \end{document} (BF₁₀ = 3.20, compared to three-main-effects model). Other main effect and interactions were not significant, Fs < 1 (BF₀₁s > 4.00). With respect to the two significant interactions, analyses of single main effects suggested that (i) restudy outperformed testing in the final recall test of list 1, p < .001, but not in the final recall test of list 2, p = .074, and (ii) restudy outperformed testing in session 2, p < .001, but not in session 1, p = .142.

Individual level

ICC estimates (and 95% confidence intervals) were calculated separately for FTE recall and intrusion scores between the two sessions. Regarding the FTE recall score, a moderate test-retest reliability was observed, ICC = .407, p = .013, 95%–CI [.052, .633]. Regarding the FTE intrusion score, no significant test-retest reliability was observed, ICC = .016, p = .475, 95%–CI [–.634, .406]. A scatterplot of the significant test-retest reliability of the FTE recall score is shown in Figure 2A.

Figure 2

List 3 recall results. (A) Scatterplot of the significant test-retest reliability of the FTE recall score, i.e., the difference between participants’ immediate list 3 recall rates in the testing and restudy condition, in Experiment 1. The red marker indicates averaged recall rates from sessions 1 and 2. Error bars show ± 1 standard error of measurement (SEM), which is the square root of the error variance term calculated from the ICC and can be interpreted as the 68% confidence interval for an individual’s data point. Color coding of data points indicates the number of subjects having identical scores on both axes. (B) Scatterplot of the nonsignificant correlation between the FTE recall and WMC scores in Experiment 2. Error bars show ± 1 standard error or the mean for the FTE recall and WMC scores, respectively. Color coding of data points indicates the number of subjects having identical scores on both axes.

Discussion

On the group level, the results of Experiment 1 replicate the findings from previous work on the forward testing effect (e.g., Bäuml & Kliegl, 2013; Szpunar et al., 2008; Weinstein et al., 2011; Yang et al., 2019), showing that interim testing of lists 1 and 2, in comparison to restudy of lists 1 and 2, enhances correct recall of list 3 items and reduced the number of prior-list intrusions in the immediate list 3 recall test. Notably, the effect of testing on prior-list intrusions was small and (Bayesian) evidence for the effect was weak. Going beyond the previous research, the results show that the forward effect on list 3 recall rates was reliably reduced in session 2 compared to session 1 (see General Discussion, for a possible explanation of this result).

On the item level, the results of Experiment 1 suggest a moderate test-retest reliability of the FTE recall score, i.e., the difference in correct list 3 recall between the testing and the restudy condition (note, however, that the 95%-CI reached from weak to substantial levels of reliability). In contrast, the reliability of the FTE intrusion score was not significant. Notably, the variance between individuals in the FTE intrusion score was very low (46 participants in session 1 and 54 participants in session 2 had an FTE intrusion score of 0; 37 participants had FTE intrusion scores of 0 in both sessions), which may have decreased the likelihood of observing a significant test-rest reliability. Arguably, a larger test-rest reliability of the FTE intrusion score may be observed with different item materials (see General Discussion).

Experiment 2

The goal of Experiment 2 was to examine the relationship between the forward testing effect and participants’ WMC. Because the reliability of a measure limits the correlation that can be observed with another measure (Nunally, 1970; Spearman, 1904), based on the results from Experiment 1, only the FTE recall score, but not the FTE intrusion score, was related to participants’ WMC. Experiment 2 took place in a single session. Participants completed the episodic memory task, including both a testing and a restudy condition, first and shortened versions of complex WM span tasks, namely the operation span and symmetry span tasks (Foster et al., 2015), second. As outlined above, no significant correlation between the forward testing effect and participants’ WMC was expected.