Background: The ε4 allele of the apolipoprotein E gene (APOE-ε4) is the strongest genetic factor for late-onset Alzheimer’s disease. During middle age, cognitively healthy APOE-ε4 carriers already show several brain alterations that resemble those of Alzheimer's disease (AD), but to a subtler degree. These include microstructural white matter (WM) changes that have been proposed as one of the earliest structural events in the AD cascade. However, previous studies have focused mainly on comparison of APOE-ε4 carriers vs noncarriers. Therefore, the extent and magnitude of the brain alterations in healthy ε4 homozygotes, who are the individuals at highest risk, remain to be characterized in detail. Methods: We examined mean, axial, and radial water diffusivity (MD, AxD, and RD, respectively) and fractional anisotropy in the WM as measured by diffusion-weighted imaging in 532 cognitively healthy middle-aged participants from the ALFA study (ALzheimer and FAmilies) cohort, a single-site population-based study enriched for AD risk (68 APOE-ε4 homozygotes, 207 heterozygotes, and 257 noncarriers). We examined the impact of age and APOE genotype on these parameters using tract-based spatial statistics. Results: Healthy APOE-ε4 homozygotes display increased WM diffusivity in regions known to be affected by AD. The effects in AxD were much smaller than in RD, suggesting a disruption of the myelin sheath rather than pure axonal damage. Conclusions: These findings could be interpreted as the result of the reduced capacity of the ε4 isoform of the APOE protein to keep cholesterol homeostasis in the brain. Because cerebral lipid metabolism is strongly related to the pathogenesis of AD, our results shed light on the possible mechanisms through which the APOE-ε4 genotype is associated with an increased risk of AD.