Background Long-term success of HIV antiretroviral therapy requires near-perfect adherence, maintained throughout one’s lifetime. However, perceptions towards ART and patterns of adherence may change during the life course. We assessed challenges to long-term adherence in adolescents and adults in three regional HIV treatment centers in Uganda. Methods We conducted 24 in-depth interviews and 2 focus group discussions with a total of 33 health-care providers and expert clients (HIV patients on long-term ART who assist with adherence support of fellow patients). Interview topics included experiences with patients on long-term treatment with either declining adherence or persistent poor adherence. Transcribed texts were coded and analyzed based on the social-ecological framework highlighting differences and commonalities between adolescents and adults. Results The overarching themes in adolescents were unstructured treatment holidays, delays in disclosure of HIV status by caretakers, stigma, which was mainly experienced in boarding schools, and diminishing or lack of clinical support. In particular, there was minimal support for early and gradual disclosure for caretakers to the infected children, diminishing clinical support for young adults during transition to adult-based care and declining peer-to-peer support group activities. The predominating theme in adults was challenges with treatment access among temporary economic migrants. Common themes to adults and adolescents were challenges with disclosure in intimate relationships, treatment related factors including side effects, supply of single tablets in place of fixed-dose combined drugs, supply of drug brands with unfavorable taste and missed opportunities for counseling due to shortage of staff. Conclusion Adherence counseling and support should be adapted differently for adolescents and adults and to the emerging life course challenges in long-term treated patients. Programs should also address constraints experienced by temporary economic migrants to ensure continuity of treatment within the host country.