Along this thesis, a novel and robust approach for automatic human annotation in long video sequences is addressed. This work defines a fully automatic pipeline that is able to deal with different types of sequences. The proposed system has been both designed and implemented following a divide and conquer approach. First, a shot detector is used to divide the sequences in smaller ones. Then, humans are detected using a face detector based on the Viola & Jones algorithm. Once humans are detected, their faces are tracked using color-based particle filters and Local Binary Patterns (LBP). Several techniques and refinements have been implemented to improve the overall robustness of the system. Moreover, a track-by-detection technique is used to enhance the tracking accuracy. Finally, each human's track is annotated throughout every shot of the sequence. The performance of the global system is assessed in experiments with real sequences and compared against human made annotations. Furthermore, these annotated tracks set the groundwork for a future recognition system, that will complete the task of automatically annotating identities throughout sequences.