The olive sector has an important social, economic, and environmental role in Spain. This production activity is distinguished by a large number of small farms, located mainly in less-favored areas (LFA). The main objective of this study is to analyze the determinant factor of farms’ technical efficiency as well as the impact of LFA payment on such efficiency. To achieve this objective, we used a Stochastic Frontier approach and a random-effect Tobit regression. A two balanced panel data sample of Spanish olive farms observed from 2000 to 2004 is obtained from the Farm Accountancy Data Network dataset. Results show that technical efficiency scores seem to be positively correlated with farm size, and age of manager, and negatively to farm renting cultivated land. The workforce composition shows that farms with a higher proportion of family labor, not receiving payments are more efficient relative to the farms with a higher proportion of remunerated work. The results also suggest a negative direct impact of the LFA aid scheme on the technical efficiency of LFA olive farms.