Magnetically doped topological insulators may produce novel states of electronic matter, where for instance the quantum anomalous Hall effect state can be realized. Pivotal to this goal is a microscopic control over the magnetic state, defined by the local electronic structure of the dopants and their interactions. We report on the magnetic coupling among Mn or Co atoms adsorbed on the surface of the topological insulator Bi2Te3. Our findings uncover the mechanisms of the exchange coupling between magnetic atoms coupled to the topological surface state in strong topological insulators. The combination of x-ray magnetic circular dichroism and ab initio calculations reveals that the sign of the magnetic coupling at short adatom-adatom distances is opposite for Mn with respect to Co. For both elements, the magnetic exchange reverses its sign at a critical distance between magnetic adatoms, as a result of the interplay between superexchange, double exchange and Ruderman-Kittel-Kasuya-Yoshida interactions.