Little is known about the patterns of sedentary time (ST) and physical activity (PA) levels throughout the week among adults and older adults with Intellectual Disability (ID). We analyzed ST and PA patterns of adults and older adults with ID. Forty-two adults and 42 older adults with mild to severe ID participated in this study. Height and weight were obtained to calculate Body Mass Index (BMI). Body fat and fat-free mass percentages were also obtained. Patterns of PA levels and ST were assessed with GT3X Actigraph accelerometers. Adults performed higher amounts of total PA and moderate to vigorous PA than older adults during the week, on weekdays and in center time (all p > 0.05). No differences between males and females were found for either PA levels or ST. Only 10.7% of the participants met the global recommendations on PA for health. The participants of the current study showed low PA levels and a high prevalence of ST. Interestingly, when comparing age and/or sex groups, no differences were observed for ST. Our findings provide novel and valuable information to be considered in future interventions aiming to increase PA levels and reduce ST.