Crossmodal semantic congruence and rarity improve episodic memory

Abstract

Semantic congruence across sensory modalities at encoding of information has been shown to improve memory performance over a short time span. However, the beneficial effect of crossmodal congruence is less well established when it comes to episodic memories over longer retention periods. This gap in knowledge is particularly wide for cross-modal semantic congruence under incidental encoding conditions, a process that is especially relevant in everyday life. Here, we present the results of a series of four experiments (total N = 232) using the dual-process signal detection model to examine crossmodal semantic effects on recollection and familiarity. In Experiment 1, we established the beneficial effects of crossmodal semantics in younger adults: hearing congruent compared with incongruent object sounds during the incidental encoding of object images increased recollection and familiarity after 48 h. In Experiment 2 we reproduced and extended the finding to a sample of older participants (50-65 years old): older people displayed a commensurable crossmodal congruence effect, despite a selective decline in recollection compared with younger adults. In Experiment 3, we showed that crossmodal facilitation is resilient to large imbalances between the frequency of congruent versus incongruent events (from 10 to 90%): Albeit rare events are more memorable than frequent ones overall, the impact of this rarity effect on the crossmodal benefit was small, and only affected familiarity. Collectively, these findings reveal a robust crossmodal semantic congruence effect for incidentally encoded visual stimuli over a long retention span, bearing the hallmarks of episodic memory enhancement.

Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This research was supported by grants from the Ministerio de Ciencia e Innovación (PID2022-137277NB-I00 AEI/FEDER) and from AGAUR Generalitat de Catalunya (2021 SGR 00911) to S.S.-F. P.A.P. was funded by a Juan de la Cierva Fellowship from the Ministerio de Ciencia e Innovación (FJC2018-037782-I).