Intra-individual Variability in Task Performance After Cognitive Training Is Associated With Long-term Outcomes in Children Part 1

Abstract

The potential benefits and mechanistic effects of working memory training (WMT) in children are the subject of much research and debate. We show that after five weeks of school-based, adaptive WMT 6–9 year-old primary school children had greater activity in prefrontal and striatal brain regions, higher task accuracy, and reduced intra-individual variability in response times compared to controls. Using a sequential sampling decision model, we demonstrate that this reduction in intra-individual variability can be explained by changes to the evidence accumulation rates and thresholds. Critically, intra-individual variability is useful in quantifying the immediate impact of cognitive training interventions, being a better predictor of academic skills and well-being 6–12 months after the end of training than task accuracy. 

Children's working memory training refers to improving children's memory and thinking abilities through a series of practice activities. Working memory is an important part of the human brain. It is responsible for processing our short-term memory and can help people better process and store information. Therefore, children must improve their working memory training.

The training process needs to be targeted at the cognitive level and age characteristics of children, especially the memory problems that children encounter in daily life so that children can use memory more efficiently in daily life.

Working memory training can be applied to many different intellectual activities, such as logical reasoning, language understanding, spatial perception, and problem-solving, to name a few. Through this training, children can improve their analytical, judgment, and problem-solving abilities.

In addition, the improvement of working memory is also closely related to children's academic performance. Learning requires us to have a strong memory ability to master knowledge. Therefore, effective working memory training can help children learn knowledge better and improve their academic performance.

In short, children's working memory training has a positive effect on children's growth. During the training process, we need to focus on understanding the children's personalities and characteristics and let them feel the happiness and sense of accomplishment that training brings to their progress. We believe that with the support of parents and teachers, children can continuously improve their thinking skills and memory through working memory training. It can be seen that we need to improve our memory. Cistanche deserticola can significantly improve memory because Cistanche deserticola is a traditional Chinese medicinal material with many unique effects, one of which is to improve memory. The efficacy of minced meat comes from the various active ingredients it contains, including acid, polysaccharides, flavonoids, etc. These ingredients can promote brain health in a variety of ways.

Click know ways to improve brain function

Taken together, our results suggest that attention control is the initial mechanism that leads to the long-run benefits of adaptive WMT. Selective and sustained attention abilities may serve as a scaffold for subsequent changes in higher cognitive processes, academic skills, and general well-being. Furthermore, these results highlight that the selection of outcome measures and the timing of the assessments play a crucial role in detecting training efficacy. Thus, evaluating intra-individual variability, during or directly after training could allow for the early tailoring of training interventions in terms of duration or content to maximize their impact.

KEYWORDS

Attention control, children, cognitive training, fMRI, intra-individual variability, working memory.

1 INTRODUCTION

Cognitive training programs have received considerable attention over the years given their potential to improve cognitive abilities in healthy and clinical populations. However, the effectiveness and persistence of benefits from cognitive training programs are still being closely examined and vigorously debated (Au et al., 2015; Bogg & Lasecki,2014; Cortese et al., 2015, Karbach & Verhaeghen, 2014; Melby-Lervag et al., 2016; Sala & Gobet, 2020, Schwaighofer et al., 2015; Smidet al, 2020, Wass et al, 2012). Although cognitive training programs have been shown to improve performance on similar untrained tasks(near-transfer), the evidence for transfer to cognitive skills in other domains (far-transfer) remains more sparse and controversial (Au et al..2015; Cortese et al., 2015, Delalande et al., 2020; Gilligan et al., 2020; Jones et al., 2020; Karbach & Verhaeghen, 2014; Melby-Lervag et al., 2016; Sala & Gobet, 2020; Schwaighofer et al., 2015; Smid et al., 2020). We still lack sufficient understanding of the types of cognitive skills and abilities that are most beneficial to train, the types of training methods and dosages that work best for particular skills, and the types of individuals that can reap sufficient benefits to justify the time and monetary costs of cognitive training interventions.

As with many aspects of cognitive training, the extent of transfer effects on academic achievements is subject to intense debate. Improvements in academic performance seem to be stronger for the language and reading domain and less consistent in mathematics, although this varies depending on the type of training regime and study sample (see Sala & Gobet, 2020 and Titz & Karbach, 2014 for review and meta-analysis). For example, there are reports that initial transfer effects to mathematics do not persist three months later (Jones et al., 2020) and that children with low working memory ability show worse math skills than a normal classroom instruction control group 2 years after working memory training (WMT) (Roberts et al., 2016). On the other hand, a recent study found that the effects of training can emerge and increase over time in a cohort of over 500 first-grade children who were not preselected based on working memory abilities (Berger et al., 2020). 

This study found that the far-transfer benefits from adaptive WMT to academic skills were only evident 6–12 months after the end of training. Moreover, this work showed that five weeks of adaptive WMT during the first-grade year led to an increased probability of entering the highest academic track of the German secondary school system 3–4 years later. Given the aggregated results across multiple studies, it is clear that longitudinal study designs that include follow-up measures over multiple years will be important for determining the potential effectiveness of different types and/or doses of cognitive training, especially for children.

It is important to understand the cognitive and neurobiological changes that take place during or just after training. Presumably, these proximal effects allow for the eventual emergence of wide-ranging benefits in the future. The level of attention can determine how well information is remembered (Gazzaley & Nobre, 2012). Working memory processes, defined as the temporary storage and manipulation of information that allows for the performance of complex cognitive tasks (Baddeley, 1996; Baddeley, 2010), are therefore closely interrelated with attention control. 

We refer to the capacity to coordinate and allocate attention to the relevant stimuli in the environment regardless of distractions and fatigue as attention control (Cartwright, 2012; Corbetta & Shulman, 2002; Norman & Shallice, 1983). Working memory requires attention control to maintain and reassess task-relevant information while blocking interference from task-irrelevant information (Engle, 2018; Fukuda & Vogel, 2011; Kane et al., 2008; Mcnab & Klingberg, 2008). Both working memory and attention control processes rely on fronto-parietal and striatal brain networks (Klingberg, 2010). Cognitive training has been reported to alter brain structure and function, with induced changes often observed in prefrontal, parietal, and striatal regions (Astle et al., 2015; Buschkuehl et al., 2012; Flegal et al., 2019; Klingberg, 2010; McNab et al., 2009; Salmi et al., 2018; Schneiders et al., 2012). 

These are crucial regions supporting executive functions such as working memory and attention control (D’Esposito & Postle, 2015; Frank et al., 2001; Mcnab & Klingberg, 2008; Owen et al., 2005; Wager & Smith, 2003). Brain imaging studies suggest that successful transfer from trained to untrained skills requires that both cognitive processes engage at least partially overlapping structural and functional brain systems (Dahlin et al., 2008; Morrison & Chein, 2011). Thus, to be most beneficial cognitive training programs should facilitate neural developments that allow for more effective and efficient engagement of such shared neural systems.

Sensitive and reliable measures of changes in mental and neural functions are necessary to detect the immediate effects of training interventions and to forecast the long-term benefits of training. In the current work, we test the hypothesis that intra-individual variability in response times may be useful in this regard. Intra-individual variability measures based on either accuracy or response times are more informative than averaged accuracy measures when trying to understand the mechanisms by which beneficial effects of cognitive training might transfer to academic skills (Karbach & Unger, 2014; Könen & Karbach, 2015), to anticipate long-term benefits in academic performance (Judd et al., 2021), and to facilitate the identification of those individuals that would benefit most from the training intervention (Karbach & Unger, 2014; Könen & Karbach, 2015; MacDonald et al., 2009; Saville et al., 2011).

Several methods have been used to quantify and distinguish between different cognitive processes that may give rise to intraindividual variability in response times. The individual coefficient of variation (ICV) is a common measure, computed as a straightforward ratio of the standard deviation relative to the mean. In addition, the shape of the response time distribution can be parameterized by fitting ex-gaussian models (Geurts et al., 2008; Hervey et al., 2006; van Belle et al., 2015), and potential sources of variability can be distinguished by fitting Diffusion Decision Models (DDM) (Forstmann et al., 2016; Karalunas & Huang-Pollock, 2013; Ratcliff et al., 2016; Schmiedek et al., 2009). Furthermore, DDMs can also be used to measure and understand the effects of attention on task performance and decision-making (Cavanagh et al., 2014; Krajbich & Rangel, 2011; Krajbich et al., 2015).

Intra-individual variability in performance is associated with the cognitive abilities and brain functions targeted by cognitive training interventions (Castellanos et al., 2005; Geurts et al., 2008; Judd et al., 2021; Kofler et al., 2013; MacDonald et al., 2006). Intra-individual variability is also associated with prefrontal brain function and dopaminergic neuromodulation (Ilg et al., 2018; Johnson et al., 2015; MacDonald et al., 2006, 2009; Papenberg et al., 2013; Tamnes et al., 2012; van Belle et al., 2015), especially the inhibitory and cognitive control abilities mediated by frontal and subcortical dopaminergic brain systems (Bellgrove et al., 2004; Isbell et al., 2018; Montez et al., 2017; van Belle et al., 2015). 

Measures of intra-individual variability may be especially useful when comparing across heterogeneous groups, such as children and adolescents whose cognitive development is still ongoing or populations with cognitive difficulties such as ADHD or Autism (Castellanos et al., 2005; Dirk & Schmiedek, 2016; Geurts et al., 2008; Karalunas et al., 2014; Könen & Karbach, 2015). Consistent with the well-established pattern of brain and cognitive development across the lifespan, intra-individual variability shows an inverted-U-shaped association with age, decreasing from childhood through adolescence until young adulthood, and increasing again in old age (Montez et al., 2017; Papenberg et al., 2013; Williams et al., 2005).

An advantage of intra-individual variability measures is that they capture not just the outcome, but also the efficiency of cognitive processes. Increased variability in response times is associated with difficulties in attention control or the ability to maintain attention and goals (Unsworth, 2015). Improved cognitive capacity, enhanced efficiency, or stability of cognitive processes have all been hypothesized as potential mechanisms underlying the impact of training interventions (von Bastian & Oberauer, 2014). Failures of attention during task performance might indicate inconsistent implementation rather than reduced capacity or abnormal function. Such inconsistency in implementing the relevant cognitive systems may not be associated with reduced accuracy if the task or test is not difficult enough and/or yields only coarse measures of accuracy, but may still significantly impair academic performance in children (Judd et al., 2021). Inconsistent implementation of behaviorally relevant cognitive processes can change the response time distribution. These effects will not necessarily lead to differences in average response times but can be quantified through various metrics of intra-individual response time variability (Ali et al., 2019; Geurts et al., 2008; van Belle et al., 2015).

Intra-individual variability has indeed been associated with attentional lapses. This has been shown in children with ADHD who often show abnormally long RTs on a subset of trials (Hervey et al., 2006; van Belle et al., 2015). Compared to measures of central tendency (e.g., mean or median), intra-individual variability measures are more sensitive to fatigue in young adults (Wang et al., 2014), or to externally rated attention problems (Gómez-Guerrero et al., 2011), and to correctly classify patients with ADHD (Klein et al., 2006). Therefore, intra-individual variability metrics could plausibly detect changes caused by training interventions that cannot (yet) be captured by mean performance measures because performance variability measures are more sensitive to the efficiency of cognitive processes.

Recent work has shown that performance variability is related to working memory abilities, training, and transfer to academic skills. Intra-individual variability in accuracy within and between sessions in working memory tasks has been associated with academic performance in 3rd and 4th-grade school children (Dirk & Schmiedek, 2016), and a recent study in 6-year-old children showed that intra-individual variability after working memory cognitive training was associated with performance in mathematics 3 years later (Judd et al., 2021). Given the predictive association between academic performance at school and well-being in adulthood (Tomasik et al., 2019), it is important to investigate the impact of any cognitive training on academic performance.

In summary, there is sufficient reason to hypothesize that intraindividual response time variability metrics can detect short-term training effects and may be useful in predicting the degree of long-term benefits. Here, we test the hypothesis that intra-individual variability in task performance—quantified via response times—can be used to assess training efficacy in the short term and is correlated with future far-transfer effects. We use a combination of cognitive tasks (N-Back and Flanker), functional magnetic resonance imaging (fMRI), and Diffusion Decision Modelling of individual performance to examine the effects of five weeks of adaptive WMT on brain and cognitive function in first-grade children. 

Our specific hypotheses were that WMT would benefit performance on the N-Back and Flanker tasks in the short term, increasing accuracy and reducing intra-individual variability in response times. We also hypothesized that decreased response time variability following WMT would be associated with brain activation in key working memory regions such as the dorsolateral prefrontal cortex and the striatum. Furthermore, we hypothesized that, if WMT influenced the ability or motivation to selectively attend to task-relevant information, then we should see differences in the estimated drift rates for the DDM between the training and control groups. Lastly, given the existing evidence for associations between intra-individual variability and cognitive function across different psychiatric and aging populations, we hypothesized that intra-individual variability measures would be indicative of future outcomes at the subsequent 6 and 12-month follow-up assessments. We test these hypotheses using three independent data sets.

2 METHOD

2.1 Participants

For this paper, we analyzed data from three separate samples of children (N = 28, 572, and 11,878). We describe the participants and tasks used in the two larger conceptual replication samples in subsection 2.7 below. The initial fMRI sample included 28 typically developing 7–9-year-old primary school children (mean age = 93 months, SD = 5 months, 14 females, working memory training group [WMT] = 16, comparison group [CMP] = 12). These children were recruited out of an ongoing intervention study of over 500 children and 29 different classrooms. The local ethics committee (Kantonale Ethikkommission Zürich) approved all procedures and methods used during this study.

2.2 Cognitive training program

The training procedures consisted of a five-week intervention and four assessment waves, one pre-intervention (baseline), one immediately after the end of the five-week intervention, and two follow-up waves at 6 and 12–13 months, respectively. The assessment battery included tests of working memory and IQ (digit span, location span, object span, Raven’s test), educational outcomes (math numeracy and math geometry, reading abilities), and concentration tests (Go/NoGo and BP task).

The working memory training program implemented was Cogmed’s RoboMemo1. It is a computerized program, highly adaptive to individual performance, implemented via notebook computers including headphones for the spoken instructions and an external mouse. The intervention consisted of a daily WMT session per day (duration ∼30 min), over 5 weeks (25 sessions). Each training session included six adaptive modules (working memory tasks), including 12 trials (75 trials in total). During the intervention, there was one specifically trained student coach in each class.

We compare the WMT group to children who either received standard classroom instruction (N = 3) or self-regulation training over six school lessons (N = 9). In these lessons, the teacher taught a version of the mental contrasting with implementation intentions (MCII) technique (Duckworth et al., 2013) that was adapted to the relevant age group and the classroom context.

2.3 Post-training cognitive and decision tasks

Working memory (N-Back) task: The 11-min block design working memory task consists of four conditions (Figure S1a). In the ‘0-Back’ condition, they have to respond whenever they see a picture of a sun on the screen. In the ‘1-Back’, ‘2-Back’, and ‘3-Back’ conditions, they have to respond whenever the picture on the screen is the same as 1, 2, or 3 before it, respectively. Performance data were recorded during scanning. The main performance variables are the d-prime index (d’ = z(HitRate) – z(FalseAlarms)) and the intra-individual coefficient of variation (ICV = SDRT/MRT) for each working memory condition.

Flanker task: The 11-minute event-related task was designed based on Rueda et al. (2004). Participants were presented with 240 trials (Figure S1b). Each trial consisted of a central row of five yellow fishes over a blue background. 

They were instructed to ‘feed’ the fish located in the center of the screen. To do so, the child had to press the right/left button on the button box, depending on the direction of the central fish and ignoring the direction of the flankers. The main performance variables are the % of correct responses and the intra-individual coefficient of variation in response times. Furthermore, we fit a decision diffusion model to the response outcomes and times to determine the source(s) of differential variability in performance (see Section 2.4.2). Lastly, for comparison with previously published papers, we also conducted a post-hoc analysis of the RT data using the ex-gaussian approach. Based on previous studies (Geurts et al., 2008; Hervey et al., 2006; van Belle et al., 2015), we expected differences between the groups in the parameters of the exponential component of the ex-gaussian distribution of RTs.

For completeness, we note that the children also completed an intertemporal choice task similar to one in Steinbeis et al. (2014) while in the scanner, which was administered to test a separate hypothesis from the one that we focus on in this paper.

Further details on the three tasks can be found in the Supplementary Methods.

2.4 Behaviour data analyses

Two of the 28 children from the fMRI sample withdrew from the study after the first task (in both cases, the intertemporal choice task). For three participants there were technical failures in collecting the performance data during the Flanker task, which resulted in one participant being excluded due to the complete loss of performance data, and for two participants only one run of the task could be used in the analysis.

2.4.1 Regressions on behavioral performance

Statistical analyses were conducted using RStudio (Version 1.1.442) (RStudio Team 2020).

We investigated differences between the trained and untrained groups on the main performance variables of the tasks right after the WMT. To do so, we conducted a general linear model (GLM) for the NBack and Flanker tasks with training group (WMT vs. CMP) as effects factor and task condition (N-Back: high vs. low working memory; Flanker: congruent vs. incongruent) as random-effects factors.

2.4.2 Decision Diffusion Modelling Analyses

We used a Bayesian hierarchical approach to fit the parameters of the decision diffusion model (DDM) to the Flanker task using JAGS (Plummer, 2003) and the JAGS-Wiener module (Wabersich & Vandekerckhove, 2014) together with the jags package (Plummer, 2018) in R. We used the priors recommend for hierarchical diffusion decision modeling in Wiecki et al. (2013). The fitting was run with three chains, 100,000 burn-in samples, and 10,000 posterior samples with a thinning rate of 10 samples. Convergence was assessed using visual inspection, and by ensuring psrf measures were below 1.05 for all parameters. Drift rates were calculated as a weighted linear combination of the target and non-target stimuli to distinguish the relative contribution of each to the evidence accumulation rate. Here, we fit the DDM to children’s behavior in the Flanker task. We did not fit data from the N-Back task because it required responses only on target trials, which were a small minority (25%) of all trials.

In our specification of the DDM, the magnitude of the drift rate coefficients informs us about how strongly each stimulus influences the evidence accumulation processes. In the flanker task, children should focus on the target fish because it alone provides evidence for the correct response in each trial. The direction the flanking distractor fish are facing is irrelevant and should be ignored. Thus, we specified the drift rate according to equation (1) below. We hypothesized that β1–β2 (i.e., the weight on relevant minus irrelevant information) should be greater in the WMT than in the CMP group.

2.5 fMRI data collection and analysis

Detailed descriptions of the fMRI preprocessing, scanning parameters, and fMRI GLM can be found in the Supplementary Methods.

2.6 Associations between post-training cognitive task performance

We investigated if intra-individual variability measures computed immediately after the training could serve as indicators of relevant future outcomes at the subsequent follow-up assessments. Specifically, we examined the total score in the SDQ (Woerner et al., 2002), a behavior and psychological well-being screening measure typically administered in clinical settings to identify potentially problematic areas in a child that merit further assessment by a specialist. The SDQ was filled out by parents 6 months after training. We also examined tests of academic performance in reading and two mathematics subscales (geometry and arithmetic) conducted 12 months after training. We focused on these specific academic skills because of the results from the independent sample in Berger et al. (2020), which show that WMT improved geometry and reading scores, but not arithmetic.

To investigate whether intra-individual variability measures could be indicative of future outcomes at the subsequent follow-up assessments across all intervention groups, we conducted Bayesian robust linear regression analyses. These analyses tested whether changes in SDQ and academic skills (i.e., controlling for baseline scores) could be explained by children’s accuracy (d-prime) or response time variability in cognitive tasks performed at the end of the training period. Specifically, we used the coefficient of variation in response times and d-prime scores from the N-Back task and the estimates of DDM drift rates from the flanker task to explain future outcomes. Certain follow-up or intra-individual variability measures were missing for some children (the maximum number of missing values for any measure was 4). To use as much of the data from our small fMRI sample as possible, we imputed the missing values using the ‘mice’ package (van Buuren & Groothuis-Oudshoorn, 2011) in R. We generated 10 different imputed datasets and fit Bayesian linear regressions to each of them using the R package, ‘brms’ (Bürkner, 2018) as an interface to STAN (Stan Development Team, 2020). We drew our final inferences from the combined posterior distributions of all ten robust regression models to reduce the influence of any one set of imputed values on our results. Each of the 10 models used z-scored dependent and independent variables and student-t priors for all fixed effects (mean = 0, SD = 1, degrees of freedom = 10). Each model used 6000 MCMC samples across three independent chains after 1000 warmup samples for each chain and a thinning step = 5. The full set of regressor variables and results from these regressions are reported in Table S7. All regressions controlled for the baseline performance for each dependent variable, which effectively estimates changes in the outcomes.

2.7 Conceptual replication and generalization using independent samples

We tested whether the core aspects would replicate or generalize in two independent, larger samples. The conceptual replication involved novel analyses of the data from Berger et al. (2020), henceforth referred to as the BFHSW study. The BFHSW study was conducted in a separate sample of 1st-grade children (age = 6–7 years, mean = 6.8 years, SD = 4.3 months) than the one from which our fMRI sample was drawn. However, the two studies used the same WMT procedures, as well as many of the same assessment instruments at baseline and follow-up. These overlaps allow us to test the association between changes in response-time ICVs after the training and their association with academic skills at the 12-month follow-up like our fMRI sample, although computing response-time ICVs from a response inhibition rather than a working memory task (see 2.7.1 for details). We also tested whether the association between intraindividual variability in task performance and measures of current and future well-being we found in our small fMRI sample would generalize to the much larger set of children taking part in the Adolescent Brain and Cognitive Development (ABCD) study (Casey et al., 2018).

Berger et al. (2020) implemented the same WMT intervention and the same pre-and post-training assessments at the same follow-up time points (6 and 12 months after the end of the training) used in our fMRI sample in a separate set of 572 children (6–7 years of age). In this sample, the WMT group included 279 participants who performed the same WMT as in our initial sample. The control groups received either standard school instruction, self-regulation training similar to our initial sample, or learning software training. Following the procedures established by Berger et al. (2020), we compared the change in ICV in response times after WMT to the 101 children who received standard school instruction. However, our primary tests of the association between changes in ICV over the five-week intervention period and improvements in reading or geometry scores 1 year after training were conducted across the 521–565 children for whom we have all of the relevant measures at each time point (see Tables 1 and 2 for details).

We computed intra-individual variability in response times from a response inhibition task instead of a working memory task in the BFHSW study. That study did not include a working memory task with enough speeded responses from each child to reliably compute intra-individual variability in response times. It did, however, include a Go/Nogo task, which measures response inhibition, at all assessment waves that we could use to compute intra-individual variability in response times. Therefore, to test the association between improvements in intra-individual variability after training and long-term, far transfer to academic skills, we computed the ICV as the standard deviation of go-trial RTs divided by the mean of the go-trial RTs. Note that we compute the ICV from correct trials only just as we did for all tasks in the initial sample and consistent with the standard procedure in the literature (Bellgrove et al., 2004; Bos et al., 2020; Fagot et al., 2018; Marciano & Yeshurun, 2017). We refer to these analyses as a conceptual replication because we use the Go/Nogo task instead of the N-back. Lastly, we only tested academic skills in the BFHSW sample because many of the parents did not complete the SDQ for their children in that study.

We fit linear regression models using Stata (StataCorp 2015). Specifically, we followed the methods reported by Berger et al. (2020) and estimated ordinary least squares regressions with robust standard errors clustered at the classroom level. All regression models included control covariates for treatment type, school, sex, age, and baseline performance in both the dependent variables (geometry or reading scores) and ICV from the Go/Nogo task.

For more information:1950477648nn@gmail.com