Background Mesial temporal lobe epilepsy (MTLE) is the most common type of focal epilepsy in adults and can be successfully cured by surgery. One of the main complications of this surgery however is a decline in language abilities. The magnitude of this decline is related to the degree of language lateralization to the left hemisphere. Most fMRI paradigms used to determine language dominance in epileptic populations have used active language tasks. Sometimes, these paradigms are too complex and may result in patient underperformance. Only a few studies have used purely passive tasks, such as listening to standard speech. Methods In the present study we characterized language lateralization in patients with MTLE using a rapid and passive semantic language task. We used functional magnetic resonance imaging (fMRI) to study 23 patients [12 with Left (LMTLE), 11 with Right mesial temporal lobe epilepsy (RMTLE)] and 19 healthy right-handed controls using a 6 minute long semantic task in which subjects passively listened to groups of sentences (SEN) and pseudo sentences (PSEN). A lateralization index (LI) was computed using a priori regions of interest of the temporal lobe. Results The LI for the significant contrasts produced activations for all participants in both temporal lobes. 81.8% of RMTLE patients and 79% of healthy individuals had a bilateral language representation for this particular task. However, 50% of LMTLE patients presented an atypical right hemispheric dominance in the LI. More importantly, the degree of right lateralization in LMTLE patients was correlated with the age of epilepsy onset. Conclusions The simple, rapid, non-collaboration dependent, passive task described in this study, produces a robust activation in the temporal lobe in both patients and controls and is capable of illustrating a pattern of atypical language organization for LMTLE patients. Furthermore, we observed that the atypical right-lateralization patterns in LMTLE patients was associated to earlier age at epilepsy onset. These results are in line with the idea that early onset of epileptic activity is associated to larger neuroplastic changes.