TECHNICAL ISO/IEC TR REPORT 29189 Firstedition 2015-06-15 Information technology Biometrics Evaluation of examiner assisted biometric applications Technologies de I'information -Biometrie Evaluation des applications biométriques assistées par un examinateur Reference number IS0/IEC TR 29189:2015(E) IE( . International Organization for Standardization dardization 5956617 @IS0/IEC 2015 stitute of Sta ted without license from IHS IS0/IEC TR 29189:2015(E) COPYRIGHTPROTECTEDDOCUMENT IS0/IEC 2015, Published in Switzerland All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form written permission. Permission can be requested from either ISO at the address below or ISO's member body in the country of the requester. ISO copyrightoffice Ch. de Blandonnet 8 . CP 401 CH-1214 Vernier, Geneva, Switzerland Tel. +41 22 749 01 11 Fax +41 22 749 09 47 [email protected] www.iso.org Intenatinair PrganizationforStandardization icene-zZhejang nsite f StandareRalS S6JEC 2015 - All rights reserved Not for Resale, 2015/9/22 08:34:18 etworking permitted without license from IHS IS0/IEC TR 29189:2015(E) Contents Page Foreword ..iv Introduction. .. 1 Scope. ..1 2 Terms and definitions 3 Symbols and abbreviated terms. 4 Example of an examiner assisted search process 5 Factors to consider when evaluating examiner assisted biometric applications. 5.1 General ..4 5.2 System-related factors to consider when evaluating examiner assisted biometric applications. .5 5.2.1 Dependencies in the flow process - Where does the examiner interact with the system?. 5.2.2 Systemandstage-levelperformancemeasurement ..5 5.2.3 Measuring 'true' operational performance 7 5.2.4 The impact of prior probabilities on human performance ..8 5.2.5 Confidence Levels.. 5.2.6 The impact of automated systems on human performance ..9 5.3 Examiner-related factors to consider when evaluating examiner assisted biometric applications. .9 5.3.1 An Examiner's perception of the system's accuracy 5.3.2 Usability and examiner acceptance ..10 5.3.3 Training and expertise. .10 5.3.4 Workload ..11 5.3.5 Bias in decision making .11 5.3.6 Individualdifferencesbetweenexaminers ..12 6 Performance evaluation of examiner assisted systems .12 6.1 Types of Evaluation.. ..12 6.2 Performancemeasures for examinerassisted biometric systems ..13 6.2.1 Introduction ..13 6.2.2 Measures of accuracy. ..13 6.2.3 Examiner-assistedperformanceconsiderationsinwatchlistscenarios .14 6.2.4 Discrimination and bias. .14 6.2.5 ExaminerDecision Confidence. .14 6.2.6 Processing speed ..14 6.3 Usability assessment. .15 6.3.1 Introduction .15 6.3.2 Qualitative observations .15 6.3.3 Questionnaires ..15 6.3.4 Interviews and focus groups .15 6.4 Reporting results. .15 6.5 Applying controls in evaluations .16 6.5.1 Introduction .16 6.5.2 Controls for examiner expertise. ..16 6.5.3 Controls for examiner decision bias ..16 6.5.4 Controls for the test environment 6.5.5 Controlsforvariationsinexaminerinput ..17 6.6 Evaluation challenges. .18 6.6.1 Introduction ..18 6.6.2 Challenges with testing on a live operational system ..18 6.6.3 Challenges in repeatable operational test ..18 Bibliography .19 iii nsee=Zhejiang Institute of Standardization 5956617 vided by IHS under lic ed without license from IHS Not for Resale, 2015/9/22 08:34:18