The Bologna Plan applied to the Medicine Degree in Spain has included a final degree dissertation into the coursework. In this manuscript, we analyzed the validity of a rubric and the grading criteria used to assess the students skills in research and their critical thinking. A total of 62 final degree dissertations were evaluated. Each student was supervised by a clinical or epidemiologist researcher. Each dissertation was rated by three academic assistants using a 10-item rubric. The validity of the scores and the differences in the severity of the examiners were analyzed with a multi-faceted Rasch model, which allowed examining severity within evaluators, ability of the students and difficulty of items. 62 final dissertation were evaluated with a total of 186 rubrics. 58% of the students had higher ability than the severity of the evaluators and the difficulty of the items of the evaluation rubric. No significant differences were observed within the severity of the professors, neither within the final scores. The most difficult item for students was selection of epidemiological design. This study showed that the methodology proposed to evaluate the final degree dissertation is effective to assess the skills of conducting research and critical thinking in medicine students.