Títol:
|
Impact of noisy annotators’ reliability in a crowdsourcing system performance
|
Autor/a:
|
Cabrera-Bean, Margarita; Diaz Vilor, Carles; Vidal Manzano, José
|
Altres autors:
|
Universitat Politècnica de Catalunya. Departament de Teoria del Senyal i Comunicacions; Universitat Politècnica de Catalunya. SPCOM - Grup de Recerca de Processament del Senyal i Comunicacions |
Abstract:
|
Crowdsourcing is a powerful tool to harness citizen assessments in some complex decision tasks. When multiple annotators provide their individual labels a more reliable collective decision is obtained if the individual reliability parameters are incorporated in the decision making procedure. The well-known Maximum A Posteriori (MAP) rule weights the individual labels in proportion to the annotators’ reliability. In this work we analyze how the crowdsourcing system performance is degraded with the use of noisy annotators’ reliability parameters and we derive an alternative MAP based rule to be applied when these parameters are neither known nor even estimated by the decision system. We also derive analytical expected error rates and their upper bounds obtained by each rule as a useful tool to estimate the number of necessary annotators in the collective decision system depending on the level of noise present in the estimated reliability parameters. |
Abstract:
|
Peer Reviewed |
Matèries:
|
-Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Processament del senyal -Signal processing -Decision-making -Crowdsource -Expected error rate bound -Specificity -Sensitivity -Tractament del senyal -Decisió, Presa de |
Drets:
|
|
Tipus de document:
|
Article - Versió publicada Objecte de conferència |
Publicat per:
|
Institute of Electrical and Electronics Engineers (IEEE)
|
Compartir:
|
|