تقرير
A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessment
| العنوان: | A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessment |
|---|---|
| المؤلفون: | Canal, Gregory, Leung, Vladimir, Sage, Philip, Heim, Eric, Wang, I-Jeng |
| سنة النشر: | 2024 |
| المجموعة: | Computer Science Statistics |
| مصطلحات موضوعية: | Statistics - Machine Learning, Computer Science - Artificial Intelligence, Computer Science - Machine Learning |
| الوصف: | Artificial intelligence (AI) has revolutionized decision-making processes and systems throughout society and, in particular, has emerged as a significant technology in high-impact scenarios of national interest. Yet, despite AI's impressive predictive capabilities in controlled settings, it still suffers from a range of practical setbacks preventing its widespread use in various critical scenarios. In particular, it is generally unclear if a given AI system's predictions can be trusted by decision-makers in downstream applications. To address the need for more transparent, robust, and trustworthy AI systems, a suite of tools has been developed to quantify the uncertainty of AI predictions and, more generally, enable AI to "self-assess" the reliability of its predictions. In this manuscript, we categorize methods for AI self-assessment along several key dimensions and provide guidelines for selecting and designing the appropriate method for a practitioner's needs. In particular, we focus on uncertainty estimation techniques that consider the impact of self-assessment on the choices made by downstream decision-makers and on the resulting costs and benefits of decision outcomes. To demonstrate the utility of our methodology for self-assessment design, we illustrate its use for two realistic national-interest scenarios. This manuscript is a practical guide for machine learning engineers and AI system users to select the ideal self-assessment techniques for each problem. |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2408.01301 |
| رقم الأكسشن: | edsarx.2408.01301 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |