Andreas Baier, Alf C. Zimmer

• Many safety-relevant tasks in control or diagnostics require binary choices such as "conflict versus separation" in air traffic control, "normal versus pathological" when interpreting x-ray pictures, or "permitted versus forbidden" when inspecting airport security scans. Deciders often are uncertain, but nevertheless required to decide between two alternatives, that is, they have not only to decide upon an action, but also about the admissible level of uncertainty. If the accepted level of judgment certainty is not taken into account, the sequence of decisions does not capture the full picture of the underlying decision process. Differences in judgment certainty are relevant, because they reflect not only the adequacy of the human-machine interface that is evaluated, but also the differences in expertise of the decider and the requirements of the actual situation or task. Therefore, capturing both judgment certainty and discrimination performance is essential. A comparison of different human-machine-interfaces (for air traffic control) is used to illustrate a methodological approach, which allows for integrated analyses of decision processes based on receiver-operator-characteristics and practical guidelines for the evaluation of human-machine-interfaces for safety-relevant operation procedures are provided.