The Limits Of Automatic Sensorimotor Processing During Word Processing: Investigations With Repeated Linguistic Experience, Memory Consolidation During Sleep, And Rich Linguistic Learning Contexts Part 3

Experiment 3

To address these issues, in Experiment 3 and 4, we moved from investigating novel words towards vertically associated familiar words whose referents participants never experienced directly (such as Hades or pterosaur). 

Thus, we employed words that are established lexicon entries and were not just recently learned in an artifcial lab setting. We can therefore assume that participants have repeatedly encountered and used these words in natural communication settings, and to have a clear meaning representation of the described entities (including their vertical location).

Methods

Participants

Following the results of the power analysis, we tested 45 participants (all right-handed; 36 female, 9 male; MAge = 23.5 years, SDAge = 6.30 years.). Data from one additional participant was excluded due to high error rates.

Materials and procedures

To create the item material, we collected rating data from 25 participants who did not participate in the actual study. For a set of 62 words, participants indicated on 5-point scales (a) the typical vertical locations associated to the word's referents (from very low to very high), and (b) how much direct experience they made with the word's referents during their lifetime (from no experience to very much experience). 

The 62 items were selected because we expected them to cover the entire range for all collected variables, including some fller items; the entire item list can be found at https://osf.io/vxrhn. Participants were explicitly instructed that depictions of the referents, for example in pictures and movies, can be counted as direct experience.

We selected eight words that were clearly associated with a vertical location, but with which participants indicated very little or no direct sensorimotor experience (see Table 2). Apart form the word material, the procedure of Experiment 3 was identical to the Test Phase of Experiment 1 and 2.

Results

The data were analyzed using the procedure described for the Test Phase analysis of Experiment 1. Error trials (3.3 %) and one overly fast trial were excluded from the analysis. Since we used real words, the factor learning context was replaced with implied location. Mean reaction times by implied location and response direction are displayed in Fig. 3 (left panel).

We performed the same model comparison as described in the previous analyses, except that the factor "learned direction" was now replaced by "implied direction". The model including the fixed effect for the two-way interaction between response direction and implied location did not perform better than the model without this interaction, as indicated by a likelihood-ratio test (휒2(1) = 1.80, p = 0.180). 

We obtained a BIC-approximated Bayes factor of BF = 0.0233, indicating that the data are about 43 times more likely under the baseline model (strong evidence in favor of the baseline model). The model parameters for the model including the interaction are reported in Table 3.

Discussion

Even though we employed real words as item material, we again observed no action-congruency efect. Thus, the factors discussed as potentially leading to the absence of this efect for novel words in Experiment 1 and 2-the limited learning experience with these novel words, the fact that participants never used them as a cue to retrieve sensorimotor experience, or that they never used or encountered them in natural contexts-do not ofer a sufcient explanation for this absence.

Interestingly, even some limited experience with the word's referents seems insufcient to elicit actioncongruency efects, as participants in the rating did not equivocally indicate that they have no experience with the referents-the ratings difer slightly from the minimum value. This raises the possibility that the action-congruency efects observed by Öttl et al. (2017) after exposing participants to the word referents were partly due to the high salience and recency of the sensorimotor experience.

Experiment 4

At this point, there is a fairly simple alternative explanation for the results in Experiment 3: Some of the words were not particularly frequent (for example Hades or supernova), and (some) participants could have simply not known the words. In this case, one could not reasonably expect any congruency efects.

We thus replicated Experiment 3, while ensuring that participants indeed knew the words presented to them. 

In this context, we also considerably extended the rating study and explicitly diferentiated between direct and "indirect" experience (for example, in pictures and movies), in order to obtain a maximally adequate item set for Experiment 4.

Method

Participants

In this experiment, we tested 44 native German speaking participants (one fewer than required due to technical problems, 41 right-handed; 35 female, 9 male; MAge = 23.6 years, SDAge = 4.24 years). Data from fve additional participants was excluded due to high error rates (see the previous experiments)

Materials and procedures

The material for Experiment 4 was obtained in a web-based rating study, using the jsPsych software (de Leeuw, 2015).

We assembled a list of 348 items, and instructed participants (who did not participate in the actual experiment) to indicate, on 5-point scales, the vertical location of the described object, the amount of direct sensorimotor experience with the described object, and the amount of "indirect" sensorimotor experience (for example in pictures or movies). They were further given the opportunity to indicate that they didn't know a word. 

The item material was selected to cover the whole range of value combinations of vertical location and amount of experience, and rating results indicated that this manipulation was successful. The entire item list can be found at https://osf.io/vxrhn.

The questionnaire was administered to 203 participants. Each participant was presented with 30 randomly selected items, resulting in between 10 and 34 ratings per word. As the up-words (down-words) for our experimental material, we selected four words that, on average, (a) received very high/very low location ratings, (b) received very low direct experience ratings, (c) received low indirect experience ratings, and (d) were known to most participants. 

Thus, the words and their referents were very familiar to participants (prevalence-the number of speakers knowing a word-is strongly correlated familiarity and word frequency; Brysbaert et al., 2019), but participants had little to no sensorimotor experience with the word referents. The selected items are displayed in Table 2. Apart from the word material, the material and procedure of the Experiment 4 test phase was identical to that of Experiment 3.

After the experiment, participants were handed a questionnaire and instructed to indicate for each of the eight words in the item material whether they knew the word or not, and if they did, the vertical location associated to the described object (up vs. down).

Results

The data were analyzed as described for Experiment 3. Again, error trials (3.3 %) and one overly fast trial were excluded from the analysis. Mean reaction times by implied location and response direction are displayed in Fig. 3 (right panel). Again, the model including a two-way interaction fxed efect between response direction and implied location did not outperform the model without this interaction (휒2(1) = 0.14, p = 0.711). 

We obtained a BIC-approximated Bayes factor of BF = 0.0103, indicating that the data are about 97 times more likely under the baseline model (strong evidence in favor of the baseline model). The model parameters for the model including the interaction are reported in Table 3. 

The pattern of results stays unchanged if we exclude from the analysis trials including words for which participants gave no answers or incorrect location judgments, or indicated that they did not know the word, in the post-test phase questionnaire (7.2% of the data).

Discussion

The results of Experiment 3 were replicated in Experiment 4, demonstrating that (a) we again fnd no evidence for an action-congruency efect for items participants have no direct sensorimotor experience with, and (b) that the absence of this efect is not due to participants not knowing the words presented to them. Interestingly, the results in Experiment 4 also indicate that low to moderate levels of indirect sensorimotor experience with the described objects are not sufcient to elicit an action-congruency effect. 

This can potentially be attributed to the fact that "depicted" objects are usually not experienced in the same vertical location as their "real" counterparts (movies and pictures are usually encountered directly in front of the observer, or on a display they are holding in their hands).

These fndings demonstrate that just controlling for the associated vertical location (see Goodhew & Kidd, 2016) when selecting the item material for studies on congruency efects is insufcient: In such studies, the absence of congruency efects could also simply result from a lack of direct experience, and not necessarily the absence of sensorimotor activation in their respective experimental paradigms.

General discussion

In four experiments, we tested whether speakers automatically activate sensorimotor experience in word processing when they have no direct experience available with the word's referents. To this end, we employed words associated with a vertical location, and employed an experimental paradigm in which previous studies observed automatic motor congruency efects during word processing (Lachmair et al., 2011; Öttl et al., 2017; Thornton et al., 2013). 

In Experiments 1 and 2, participants learned novel words and their associated vertical locations in several separate learning phases, with an intervening memory consolidation via night's sleep. 

In Experiment 3 and 4, we employed familiar, vertically-associated words whose referents participants did not experience directly. In line with previous results by Günther et al. (2018), we did not observe automatic action-congruency efects for words learned from language alone in any of the experiments, even though participants in the present set of studies made far more experience with the words in richer linguistic learning contexts, and even though they had the opportunity for memory consolidation during sleep.

Discussion of alternative explanations

This pattern of results is not a consequence of using only a limited number of eight items: In a pilot experiment, Günther et al. (2018) observed an action-congruency efect for a set of eight real words for which direct experience available (cloud or basement), the same number as items in the present experiments. It is also not a consequence of using novel words per se: Öttl et al. (2017) observed the efect for their set of eight novel words for which direct experience with the referents was available. 

Furthermore, Günther et al. (2020) also observed a congruency efect using a set of eight words, both for real words (Experiment 1) as well as novel words (Experiment 2). 

By following the results of an earlier power analysis (see Günther et al., 2018), it is also unlikely that the absence of an efect in four diferent experiments results from insufcient statistical power (even if the power estimate of 0.90 was an extreme over-estimation and the true power of each of our experiments was only at 0.50, the probability of not fnding an existing efect would only be at (1 − 0.50) 4 = 0.0625 across all four studies). 

Finally, the absence of an efect can also not be ascribed to participants not understanding the words or not associating them with a vertical dimension, as indicated by the explicit judgment task in Experiment 1 and 2 and the rating results in Experiment 3 and 4.

In principle, there is also the possibility that we did not fnd a congruency efect as a result of the specifc properties of the item material and the required response: In all experiments, participants had to react with upwards or downwards hand movements, while interactions with the referents of the presented words would not necessarily involve such vertical hand movements (take, for example, an underground city or artifcial sun in Experiments 1 and 2, or earth's core or Jupiter in Experiments 3 and 4). 

However, previous studies have shown that this word-level congruency efect is also consistently found for items such as plateau, planet, sky, cloud, or skyscraper on the one hand, and swamp, submarine, basement, or underground on the other hand (Lachmair et al., 2011)-all entities that we arguably don't interact with more using vertical hand movements than the concepts in question. 

We thus don't consider it likely that specifcally this property of the item material used in the present studies caused the absence of the efect. However, it might still be the case that automatic congruency efects are more likely for non-experienced concepts that would inherently aford such vertical movements, which can be investigated in future studies.

In the context of this argument, it is also important to note that the congruency efect investigated here is not the classical action-sentence congruency efect (ACE; Glenberg & Kaschak, 2002) found for sentences describing certain actions and specifc movements; instead, it is a pure word-level efect. This is especially relevant as the reliability of the classical ACE has recently been called into question (Papesh, 2015), especially since a large multi-lab collaboration has failed to replicate it (Morey et al., in press). 

However, this debate on the ACE did not yet consider word-level efects, which have been reliably observed across many different studies (Dudschig et al., 2012, 2014a, b; Dudschig & Kaup, 2017; Lachmair et al., 2011; Öttl et al., 2017; Thornton et al., 2013; Vogt et al., 2019; see also the pilot study in Günther et al., 2018). In the studies where this word-level efect was not observed, this can either be attributed to missing saliency of the vertical dimension in both the stimulus and response set (Dudschig & Kaup, 2017) or, as in the studies presented here, to the specifc word material (novel word labels for non-experienced referents; compare Günther et al., 2018). 

Given the apparent robustness of this word-level efect,4 we don't consider it likely that the null results of the present study are the result of a general non-replicability.5

Theoretical implications

Our results are in line with accounts postulating that language processing does not always entail the automatic activation of sensorimotor experience (see Lebois et al., 2015). This is often explained in terms of task demands, in that we only engage in sensory and motor processing when required by the task (Günther et al., 2020; Ostarek & Huettig, 2019). However, we need to extend upon this explanation: As demonstrated by previous studies, such automatic actioncongruency efects also emerge when direct experience with word referents is available, irrespective of whether they are well-known familiar words (Lachmair et al., 2011) or newlylearned novel words (Öttl et al., 2017). 

On the other hand, when that direct experience is missing, we observe no such efects, neither for novel (Experiment 1 and 2) nor for familiar words (Experiment 3 and 4). Thus, sensorimotor experience can be automatically activated even when not required by the task, but only if direct experience with the referent is available and sufciently strong links to the linguistic stimulus are established.

Taken together, we can thus identify factors leading to activation of sensorimotor experience during language processing. Previous research has shown that concepts can be linked to sensorimotor experience (Lachmair et al., 2011), and that these connections can be established directly through experience with the referent (Öttl et al., 2017), but also indirectly via language (Günther et al., 2020). 

Whether this information is activated in a given context then depends on to what extent this "sensorimotor schema" is made salient. In cases where the connection to sensorimotor experience is strongly established, which can for example result from direct referent experience, this information is salient by default and will thus be easily activated even if not required by the task (as in the original Stroop task; Stroop, 1935). However, even then the task (i.e., the context in which language processing takes place) has to make this "sensorimotor schema" at least minimally salient: When reducing the saliency of the vertical dimension in the item set (by including words not related to a vertical dimension) as well as the response set (by including horizontal in addition to vertical answers), the action-congruency efect observed by Lachmair et al. (2011) disappears (Dudschig & Kaup, 2017). 

On the other hand, even in cases where the connection to sensorimotor experience is weaker-for example, when direct referent experience is missing-it can still be made salient depending on the task at hand and the level of processing: Günther et al. (2020) observe action-congruency words for novel words learned from language alone in a plausibility judgment task for sentences, which necessarily requires meaning access and a simulation of the sentence contents. However, as demonstrated in the present study, indirect connections between words and sensorimotor experience provided via language are not by themselves salient enough to be spontaneously activated.

In principle, it might of course still be the case that the availability of direct referent experience is not the deciding factor at play here. For example, one could assume that sensorimotor simulations can play the role of actual direct experience: If participants consistently had to simulate actions including the newly-learned words, such as You scratch your mende when they learned that a mende is a bionic foot, these simulations could be sufcient to establish strong connections between the words and sensorimotor information. 

In fact, judging the plausibility of such sentences was the test phase employed by Günther et al. (2020). If we imagine an experimental setup where such plausibility judgments form the learning phase instead of the test phase, one might possibly expect to observe the automatic congruency efects that were absent in the present study. We leave such investigations to future research. 

Nevertheless, from present study we can conclude that neither (a) a memory consolidation phase via sleep nor (b) a rich linguistic learning context result in any measureable automatic activation of the sensorimotor meaning aspect of words learned from purely linguistic experience.

Acknowledgements We thank Emmanuel Vrachimis for his contribution to the construction of the item material, and our student assistants for their help in the data collection. All data, analysis scripts, and material are available at https://osf.io/vxrhn.

Funding Open Access funding enabled and organized by Projekt DEAL.

Declarations

Conflict of interest All authors have no conficts of interest to declare that are relevant to the content of this article.

Ethical approval This study was funded by a Research Fellowship (no. 392225719) from the German Research Foundation (DFG), awarded to Fritz Günther, and by the Collaborative Research Centre 833 (SFB 833) "The Construction of Meaning"/Z2 project appointed to Sigrid Beck and Barbara Kaup by the DFG. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards, and approved by the department's ethical comittee (Kommission für Ethik in der psychologischen Forschung), Az: Kaup_2018_0907_135. Informed consent was obtained from all individual participants included in the study.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. 

The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

References

Baayen, R. H., & Milin, P. (2010). Analyzing reaction times. International Journal of Psychological Research, 3(2), 12–28.

Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random efects structure for confrmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68, 255–278.

Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22, 637–660.

Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-efects models using lme4. Journal of Statistical Software, 67(1), 1–48.

Borghi, A. M., Glenberg, A. M., & Kaschak, M. P. (2004). Putting words in perspective. Memory & Cognition, 32, 863–873.

Brainard, D. H. (1997). The psychophysics toolbox. Spatial Vision, 10, 433–436.

Brysbaert, M., Mandera, P., McCormick, S. F., & Keuleers, E. (2019). Word prevalence norms for 62,000 english lemmas. Behavior Research Methods, 51, 467–479.

Brysbaert, M., Stevens, M., Mandera, P., & Keuleers, E. (2016). How many words do we know? Practical estimates of vocabulary size dependent on word defnition, the degree of language input and the participant's age. Frontiers in Psychology, 7, 1116.

Collins, A. M., & Loftus, E. F. (1975). A spreading-activation theory of semantic processing. Psychological Review, 82, 207–428.

de Leeuw, J. R. (2015). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods, 47, 1–12.

Dudschig, C., de la Vega, I., De Filippis, M., & Kaup, B. (2014). Language and vertical space: On the automaticity of language action interconnections. Cortex, 58, 151–160.

Dudschig, C., de la Vega, I., & Kaup, B. (2014). Embodiment and second-language: Automatic activation of motor responses during processing spatially associated L2 words and emotion L2 words in a vertical Stroop paradigm. Brain and Language, 132, 14–21.

For more information:1950477648nn@gmail.com