Quality Control in Crowdsourcing

Home
Information and Communication
Quality Control in Crowdsourcing

Information and Communication

Contact 日本語

Satoshi Oyama Associate Professor

Faculty of Information Science and Technology

Hokkaido University Researcher Directory

Highly accurate decision-making using people’s confidence rating

We are conducting research to guarantee the quality of work results in crowdsourcing, which allows us to commission jobs to many people via the Internet. The use of workers’ confidence rating on the work results will be effective to ensure high-quality work results.

Content of research

With the advent of crowdsourcing services in recent years, it has become easy to commission jobs (tasks) to a large number of people via the Internet, and these services are being used in various fields of information science (image recognition, natural language processing, information retrieval, databases, etc.). In crowdsourcing, it is important to check the work quality because not all workers necessarily have the required skills and diligence to work on a task. We have proposed a method to ensure the work quality by asking workers to report their confidence (degree of conviction) in their work results. The technical feature of this method is that it does not trust the confidence rating reported by workers as it is, but performs statistical quality control assuming the existence of over- and under-confident workers.

Conceptual diagram of crowdsourcing services. Crowdsourcing makes it possible to offer jobs to a large number of people through the Internet.
Relationship between confidence rating and actual percentage of correct answers. Since there are over- and under-confident workers, it is necessary to consider the discrepancy in the accuracy of workers’ confidence rating for quality control.

Potential for social implementation

・Preparation of training data (correct answer data) necessary to realize image classification, text classification, etc. using machine learning
・Realizing highly reliable decision-making in marketing, etc. based on the aggregated opinions of multiple people

Appealing points to industry and local governments

Although crowdsourcing is a relatively new form of service, it has the potential to be applied to a variety of tasks in companies. We hope to conduct joint research on quality control, which tends to be an issue when companies use crowdsourcing for specific tasks.