HIV controllers (HICs) spontaneously maintain HIV viral replication at low level without antiretroviral therapy (ART), a small number of whom will eventually lose this ability to control HIV viremia. The objective was to identify factors associated with loss of virological control. HICs were identified in COHERE on the basis of ≥5 consecutive viral loads (VL) ≤500 copies/mL over ≥1 year whilst ART-naive, with the last VL ≤500 copies/mL measured ≥5 years after HIV diagnosis. Loss of virological control was defined as 2 consecutive VL >2000 copies/mL. Duration of HIV control was described using cumulative incidence method, considering loss of virological control, ART initiation and death during virological control as competing outcomes. Factors associated with loss of virological control were identified using Cox models. CD4 and CD8 dynamics were described using mixed-effect linear models. We identified 1067 HICs; 86 lost virological control, 293 initiated ART, and 13 died during virological control. Six years after confirmation of HIC status, the probability of losing virological control, initiating ART and dying were 13%, 37%, and 2%. Current lower CD4/CD8 ratio and a history of transient viral rebounds were associated with an increased risk of losing virological control. CD4 declined and CD8 increased before loss of virological control, and before viral rebounds. Expansion of CD8 and decline of CD4 during HIV control may result from repeated low-level viremia. Our findings suggest that in addition to superinfection, other mechanisms, such as low grade viral replication, can lead to loss of virological control in HICs