Human uncertainty in interaction with a machine : establishing a reference dataset

We investigate the task of malformed object classification in an industrial setting, where the
term ‘malformed’ encompasses objects that are misshapen, distorted, corroded or broken.
Recognizing whether such an object can be repaired, taken apart so that its components can be used otherwise, or dispatched for recycling, is a difficult classification task. Despite the progress of artificial intelligence for the classification of objects based on images, the
classification of malformed objects still demands human involvement, because each such
object is unique. Ideally, the intelligent machine should demand expert support only when it is uncertain about the class. But what if the human is also uncertain? Such a case must be
recognized before being dealt with. Goal of this research thread is to establish a reference
dataset on human uncertainty for such a classification problem and to derive indicators of
uncertainty from sensory inputs. To this purpose, we designed an experiment for an object
classification scenario where the uncertainty can be directly linked to the difficulty of
labelling each object. By thus controlling uncertainty, we intend to build up a reference
dataset and investigate how different sensory inputs can serve as uncertainty indicators for
these data.