Part 1:Goal-Directed Modulation Of Neural Memory Patterns: Implications For FMRI-Based Memory Detection
Mar 19, 2022
Contact: Audrey Hu audrey.hu@wecistanche.com
Melina R. Uncapher,1* J. Tyler Boyd-Meredith,1* Tiffany E. Chow,3 Jesse Rissman,3 and X Anthony D. Wagner1,2
1Department of Psychology and 2Neurosciences Program, Stanford University, Stanford, California 94305, and 3Department of Psychology, University of California, Los Angeles, Los Angeles, California 90095
Remembering a past event elicits distributed neural patterns that can be distinguished from patterns elicited when encountering novel information. These differing patterns can be decoded with relatively high diagnostic accuracy for individual memories using multi-voxel pattern analysis (MVPA) of fMRI data. Brain-based memory detection—if valid and reliable—would have clear utility beyond the domain of cognitive neuroscience, in the realm of law, marketing, and beyond. However, a significant boundary condition on memory decoding validity may be the deployment of “countermeasures”: strategies used to mask memory signals. Here we tested the vulnerability of fMRI-based memory detection to countermeasures, using a paradigm that bears resemblance to eyewitness identification. Participants were scanned while performing two tasks on previously studied and novel faces:(1)standard recognition memory task; and (2) a task wherein they attempted to conceal their true memory state. Univariate analyses revealed that participants were able to strategically modulate neural responses, the average across trials, in regions implicated in memory retrieval, including the hippocampus and angular gyrus. Moreover, regions associated with goal-directed shifts of attention and thought substitution supported memory concealment, and those associated with memory generation supported novelty concealment. Critically, whereas MVPA enabled reliable classification of memory states when participants reported memory truthfully, the ability to decode memory on individual trials was compromised, even reversing, during attempts to conceal memory. Together, these findings demonstrate that strategic goals Tate can be deployed to mask memory-related neural patterns and foil memory decoding technology, placing a significant boundary condition on their real-world utility.
Keywords: countermeasures; episodic retrieval; functional MRI; neurolaw; pattern classification
Introduction
Growing evidence indicates that it is possible to decode the presence or absence of memory for a stimulus or event from distributed patterns of human brain activity, as measured by functional MRI (fMRI) and multivoxel pattern analysis (MVPA; Johnson et al., 2009; McDuff et al., 2009; Chadwick et al., 2010; Quamme et al., 2010; Rissman et al., 2010; Polyn et al., 2012; Poppenk and Norman, 2012; Rissman and Wagner, 2012). The rapidly emerging literature on fMRI-based memory decoding not only informs neurocognitive theories of memory but also has implications for law, marketing, and beyond (Meegan, 2008). For example, a re- liable and validated method to detect memory could advance the forensic ability of the criminal justice system to determine whether a suspect has guilty knowledge of crime-relevant information(Greely,2011)or whether an eye witness recognizes a critical event element. Given the high diagnostic accuracy observed in some fMRI-based memory decoding studies (up to 70–90%; Rissman et al., 2010), it may be tempting to conclude that these approaches have forensic utility for uncovering an individual’s memory states and perhaps also their experiential history with event information.
However, fMRI-based memory detection techniques are still under development, with many significant challenges remaining before determining their appropriateness for field use (Brown and Murphy, 2010; Verschuere et al., 2011). One of the most significant open questions is whether memory decoding is vulnerable to“countermeasures”: strategies deployed to mask mem or signals and“beat” detection tests (Farah et al.,2014). Rissman et al. (2010)reported indirect evidence suggesting a vulnerability to strategic goal states, because the ability to detect previously encountered from novel faces was reduced to near chance when participants’ memory was implicitly, rather than explicitly, probed. However, other data suggest that lack of attention to one’s mnemonic state may not always thwart memory classification. Forexample, Kuhletal. (2013) was able to decode memory details even when participants were not instructed to retrieve those details. Together, these findings reveal a need to identify conditions under which strategic goal states alter neural memory patterns: in particular, can participants willfully conceal their memory states through the use of countermeasures that appear cooperative? Addressing this question not only has implications for delineating the boundary conditions of fMRI methods to detect memory but also for understanding the dynamics of goal-directed retrieval processes.
Here we investigated a situation that resembles eyewitness identification and required countermeasures that would appear cooperative on an eyewitness identification test. Participants viewed a series of faces, and their memory for these faces was then probed in one of two ways while undergoing fMRI. In the first test, participants made explicit recognition decisions about previously encountered and novel faces. In the second test, participants attempted to conceal their memory for the previously encountered faces and to feign memory for the novel faces. Using the explicit memory data, we trained classifiers to dis- criminate activity patterns associated with the subjective experiences of recognition and novelty. We then tested whether the classifiers could decode the participants’ memory states when they engaged in countermeasures.
Materials and Methods
Participants. Twenty-four healthy, right-handed male participants were recruited from Stanford University and its surrounding communities. Partic- ipantswereaged18–31years, with a mean SDageof23 4.29years, were native English speakers with no history of neurological complications, and were either African American (AA; n 8) or European American (EA; n 16) according to self-report. Participants gave written informed consent, in accordancewithStanfordUniversityInstitutionalReviewBoardprocedures, and were screened for fMRI compatibility.
Experiment. The experiment included two scan sessions conducted 24 h apart and took approximately 5 h across both scan sessions. Each participant was compensated $20 for each hour of participation. Data from two additional participants were collected but omitted from subsequent analyses because of inadequate or incomplete performance: one was omitted because d was at chance ( 0.08) and the other because the participant withdrew from scanning before completing the experiment.
Stimuli. Face stimuli consisted of 400 color photographs of male faces, of which half were AA and half were EA (data examining the effects of the race will be reported separately). Face stimuli were standardized for neutral facial expression and background illumination and included head and neck only. Stimuli were presented against a gray background with a black central fixation crosshair. For each participant, face stimuli were divided into two samples using stratified random sampling by race to assign stimuli to be presented during the encoding phase (OLD items; 100AA,100EA)or to serve as foil items at retrieval(NEWitems;100AA, 100 EA).
Day 1: encoding. Participants were scanned while intentionally encoding 200 male faces (100 AA faces and 100 EA faces). Each face was presented for 2 s, with an 8 s interstimulus interval (ISI) for a total of 10 s per trial. Each face was shown twice during the course of the encoding phase: after the full set of 200 stimuli was presented, the same faces were presented again in a different order. Participants were given an elaborative encoding strategy to memorize the stimuli, whereby they were instructed to generate imaginative stories involving the individuals pictured in the stimuli. To confirm that participants were attending to stimuli and engaging in the task, they were instructed to press the right index finger button on a response box after each face appeared. Stimuli were presented across eight runs, with 50 faces per run. The first four runs consisted of the first presentation of study stimuli, and the second four runs consisted of the second presentation of study stimuli. At the end of each run (for both day 1 and day 2), participants were encouraged to take as long a break as they desired, self-terminating the break via button press. Breaks ranged between 10 and 55 s across participants.
Specifically, for faces thought to be studied previously, participants were instructed to indicate a “new” response and to rapidly switch to a cognitive strategy to conceal the neural correlates of their recognition: they were asked to focus on the technical and/or photographic aspects of the stimulus to which they had not attended previously, such as the exposure, lighting, line edges, etc. Here, it was stressed that the computer algorithm was sensitive enough to pick up memory signals and that, after determining that it was a studied face, they should—to the best of their ability—try to keep the memory for the face from coming to mind by attending to the perceptual aspects of the photographs. Conversely, for stimuli perceived to be unstudied, participants were instructed to indicate an “old” response and to use cognitive strategies to elicit the neural correlates of retrieval, i.e., bring to mind a known individual that resembled the face, and relive any memories associated with the individual. Again, it was stressed that the computer algorithm was sensitive enough to detect novelty signals, so they should rapidly switch to generating memories for the face they determined to be novel. For both old and new faces, participants were instructed to use the appropriate cognitive strategy for the entire duration of the trial (10 s). Experimenters confirmed that all participants understood the concealed memory task before proceeding with the experiment.
Univariate fMRI analyses. Statistical Parametric Mapping (SPM8; Wellcome Department of Cognitive Neurology, London, UK; http:// www.fil.ion.ucl.ac.uk/spm/software/spm8), run in MATLAB 7.7 (R2008b; MathWorks), was used for both data preprocessing and uni- variate analysis.
Standard preprocessing procedures were applied to the data. All functional volumes were slice-time corrected to account for acquisition time differences between slices, with the middle slice in time used as a reference. All functional volumes were motion-corrected and spatially realigned to the first volume, followed by realignment to the mean volume of the session. The T2-weighted anatomical volume from the day 2 (retrieval) session was coregistered to the mean functional volume, the T1- weighted anatomical volume was then coregistered to this coregistered T2-weighted volume, and then the T1-weighted volume was segmented into gray matter, white matter, and CSF, with the resulting images normalized to templates in Montreal Neurological Institute (MNI) space. Functional volumes were normalized into standard space based on the transformation parameters obtained during segmentation and resampled into 4 mm 3 voxels. All images were then spatially smoothed with an 8 mm full-width at half-maximum (FWHM) Gaussian kernel.
For all classification schemes, trial counts were balanced across classes (via random subsampling) within the training and testing bins before classification to ensure a theoretical null hypothesis classification accuracy rate of 50% and an area under the curve (AUC; see below) of 0.50; analyses with shuffled class labels confirmed that chance classification performance converged around these levels (“null distribution”). After balancing, the data from each voxel were z scored again, such that the mean activity level of each voxel for Class A trials was the inverse of its mean activity level for Class B trials. For each analysis, the entire classification process was run 10 times to obtain stable estimates of performance (independent analyses confirmed that 10 iterations were sufficient to obtain stable performance estimates).
Regularized logistic regression (RLR) was used for all classification procedures. This was determined previously to be an advantageous choice in this classification paradigm byRissman et al. (2010). This algorithm implemented a multiclass logistic regression function using a softmax transformation of linear combinations of the features (Bishop, 2006) with an additional ridge penalty term as a Gaussian prior to the feature weights. This penalty term provided L2 regularization, enforcing small weights. During classifier training, the RLR algorithm learned the set of feature weights that maximized the log-likelihood of the data; feature weights were initialized to zero, and optimization was implemented with Carl Rasmussen’s conjugate gradient minimization function (http://www.gatsby.ucl.ac.uk/ Edward/code/minimize/) using the gradient of the log-likelihood combined with the L2 penalty.
The L2 penalty was set to be half of the additive inverse of a user-specified parameter, multiplied by the square of the L2 norm of the weight vector for each class, added over classes. We elected to set this free parameter to a fixed value of 10 for all analyses reported in this study.
Assessingclassifierperformance.AfterfittingtheRLRmodelparameters using the training set data, each brain activity pattern (i.e., trial) from the test set was then fed into the model and yielded an estimate of the probability of that example being from Class A or Class B (by construction, these two values always sum to one). These probability values were concatenated across all cross-validation testing folds and then ranked. The true positive (hit) rate and false positive (FA) rate of the classifier were calculated at 80 fixed cutoff thresholds along the probability continuum to generate receiver operating characteristic (ROC) curves. The AUC values associated with these curves were computed as described by Fawcett (2004) and can be interpreted formally as the probability that a randomly chosen member of one class has a smaller estimated probability of belonging to the other class than has a randomly chosen member of the other class. Stated another way, the AUC indexes the mean accuracy with which a randomly chosen pair of Class A and Class B trials could be assigned to their correct class (0.5 is random performance; 1.0 is a perfect performance). If one’s goal is high specificity in labeling examples of Class A and is unwilling to tolerate many false positives, one can interrogate the most confident guesses of the classifier. Here we arbitrarily set this threshold to be the top 10% of classification guesses. Note that we report accuracy rather than AUC values when reporting the most confident trials of the classifiers.
Importance maps. For each classification scheme, importance maps were constructed following the procedure described in previous MVPA studies (Johnson et al., 2009; McDuff et al., 2009). The importance value of a voxel provides an index of how much its signal increases or decreases influence the predictions of the classifier. After training, the logistic regression classification procedure yields a set of weight values reflecting the predictive value of each voxel (with positive values indicating that activity increases are generally associated with a Class A outcome and negative values indicating that activity increases are generally associated withaClassBoutcome).Theseweightswerethenmultipliedbythemean activity level of each voxel for Class A trials (which, because of our trial balancing and z-scoring procedure, is the additive inverse of its mean activity level for Class B trials). Voxels with positive values for both activity and weight were assigned positive importance values, voxels with negative activity and weight were assigned negative importance values, and voxels for which the activity and weight had opposite signs were assigned importance values of zero (Johnson et al., 2009; McDuff et al., 2009). Group-level summary maps were created by averaging the importance maps of the individual participants and are displayed in the figures at arbitrary thresholds: 3D-rendered maps thresholded between 0.02 and 0.5 and 2D-rendered maps between 0.05 and 0.5 (seeFig. 4) or between 0.15 and 0.5 (seeFig. 7). As a final note, although importance maps are a useful tool to evaluate which voxels were used by the classifier, these maps should not be interpreted as providing an exhaustive assessment of which voxels are individually informative about the distinction of interest.
Searchlight analyses. Importance maps reveal which voxels provide diagnostic information to the whole-brain classifiers. However, they do not reveal whether data from individual anatomical regions can be used on their own to discriminate hits from CRs. We conducted searchlight analyses to provide local decoding accuracies (Kriegeskorte et al., 2006) across the brain. Of particular interest was whether the regions in which mean blood oxygen level-dependent (BOLD; univariate) signal was modulated significantly by countermeasures (seeFig. 2A) also enabled trial-by-trial decoding accuracy that significantly departed from chance. We performed the critical classification (explicit ¡ concealed hits vs CRs) on local spherical masks centered individually on every voxel in the whole-brain mask (excluding voxels in the motor cortex and cerebellum). Each spherical mask included any voxel that touched the edge of the center voxel; thus, the resulting spheres contained 19 voxels, except when the sphere extended beyond the whole-brain mask. To determine whether local decoding accuracies evolved across the trial (as would be expected if participants initially attended to memory signals and then attempted to conceal such signals), we conducted these searchlights separately for each of the six TRs.
We evaluated significance in each of our searchlight spheres as in the prior decoding analyses: AUCs were first computed for 10 classification iterations, and then a null distribution was simulated by computing 10 additional classification iterations using scrambled regressors. We generated group-level t maps showing spheres that reliably discriminated hits from CRs by performing a paired t-test of each participant’s mean scrambled versus unscrambled AUC value at each voxel, across all 10 iterations. These maps were thresholded at p 0.05 (corrected) by applying a cluster-size threshold derived from Monte Carlo simulations (Xiong et al., 1995) as implemented in the AFNI (Automated Functional Neuro-Imaging) program 3dClustSim. The smoothness of the Monte
Carlo simulation was estimated separately for each participant and each time point using the AFNI program 3dFWHMx from the average AUCs achieved across the iterations of scrambled classification. Smoothness was averaged across participants and time points to compute a single smoothness value for each dimension. A voxelwise height threshold of p 0.01 resulted in a cluster size of 22 voxels to reach a cluster-level significance of p 0.05 (FWE) within a given time point. To correct for multiple comparisons across our six-time points, we applied a Bonferroni- ni’s correction, computing the extent threshold necessary to achieve a cluster-level significance of p 0.0083 (or 0.05/6; FWE) at each time point or p 0.05 (FWE) across space and time. Using this method, we determined that a cluster extent of 29 voxels was required to achieve a cluster-level significance of p 0.05 (FWE) across space and the six-time points.
Results
Behavioral performance
Explicit memory task
When participants truthfully reported their mnemonic experience elicited by each test face, they achieved a mean SD hit rate (the rate at which OLD images accurately judged “old”) of 0.73 0.12 and an FA rate (the rate at which NEW images inaccurately judged “old”) of 0.27 0.10, resulting in a mean d of 1.27 0.56. Mean response times (RTs) were faster for correct answers (hits, 1.74 0.60 s; CRs, 2.10 0.75 s) than for incorrect answers (FAs, 2.30 1.01 s; misses: 2.33 0.81 s; t(23) 5.15, p 3.22 10 5). Hit responses were faster than CR responses (t(23) 5.44, p 1.56 10 5).
Comparing explicit and concealed memory tasks
Mean d was significantly greater in the explicit memory than in the concealed memory task (t(23) 2.99,p 6.6 10 3). There were no task differences in mean RT for any memory outcome (all p values 0.05). However, there was a significant interaction between task and memory,inthattheaveragedifferenceinRTfor hits and CRs was greater in the explicit memory than in the concealed memory task (t(23) 6.25, p 2.22 10 6). This differential effect of memory on RT as a function of the task may follow from the difference in d between the explicit and concealed memory conditions and is likely a consequence of the dual-task nature of the concealed memory condition: participants were required to first determine whether faces were old or new and then rapidly switch to a memory/novelty concealment strategy while also reversing their motor responses.
Univariate fMRI analyses
We first investigated the question of whether participants could engage in strategic countermeasures to modulate memory-related BOLD signals across trials (i.e., univariate fMRI responses). To do so, we identified “memory success effects” (hits CRs) for each task separately and then determined where memory success effects were common across tasks, as well as where they were modulated by task. We then investigated whether the ability to modulate memory success effects was influenced by memory strength.
Common memory success effects
Given that participants were required to determine whether a face was old or new in both tasks, we next sought to determine whether there were any regions that differentiated hits from CRs in both tasks. To do so, we inclusively masked the foregoing memory success contrasts(at p 0.01each, to result in a conjoint threshold of p 0.001). The outcome of this masking procedure revealed effects in the left IPS and left VTC (Fig. 1C).
lesslikelytoshowreversedmemorysuccesseffectsintheleftAnG (r 0.36, p 0.04). In other words, their memory success effects persisted despite attempts to conceal their memory. This finding suggests that participants with stronger memories had greater difficulty modulating their AnG activity during the concealed task. Interestingly, neither hippocampal cluster showed a significant relationship between memory strength and activity (right hippocampus,r0.05,p 0.82; left hippo campus,r0.22, p 0.29), and the slopes of the correlations differed between the AnG and right hippocampus (Williams t(21) 2.49,p 0.021) and marginally differed for the left hippocampus (Williams t(21) 1.62, p 0.12). Together, these findings suggest that participants with stronger memories were less able to exert goal-directed control over memory-relatedactivityintheleftAnG, but this appeared not to be the case in the bilateral hippocampus.
Multivariate fMRI analyses
Our central question is whether use of cognitive (goal-directed) countermeasure strategies would enable participants to mask neural patterns related to memory, thus affecting the ability of multivariate techniques to read out their memory states for individual events. Accordingly, we next assessed the ability of MVPA classifiers to decode the memory status of individual retrieval trials by (1) first training and testing a classifier on data from the standard recognition memory task (explicit memory task) and (2) then assessing whether this classifier could also decode memory when participants were attempting to conceal their memory states (concealed memory task). Our process model suggested three alternative scenarios to test; we begin by explicating our process model and then describe each hypothesis in turn.
