Cyber & IT Supervisory Forum - Additional Resources

Measure 4.3 Measurable performance improvements or declines based on consultations with relevant AI actors including affected communities, and field data about context-relevant risks and trustworthiness characteristics, are identified and documented. About TEVV activities conducted throughout the AI system lifecycle can provide baseline quantitative measures for trustworthy characteristics. When combined with results from Measure 2.5 to 2.11 and Measure 4.1 and 4.2, TEVV actors can maintain a comprehensive view of system performance. These measures can be augmented through participatory engagement with potentially impacted communities or other forms of stakeholder elicitation about AI systems’ impacts. These sources of information can allow AI actors to explore potential adjustments to system components, adapt operating conditions, or institute performance improvements. Suggested Actions Develop baseline quantitative measures for trustworthy characteristics. Delimit and characterize baseline operation values and states. Utilize qualitative approaches to augment and complement quantitative baseline measures, in close coordination with impact assessment, human factors and socio-technical AI actors. Monitor and assess measurements as part of continual improvement to identify potential system adjustments or modifications. Perform and document sensitivity analysis to characterize actual and expected variance in performance after applying system or procedural updates. Document decisions related to the sensitivity analysis and record expected influence on system performance and identified risks. Transparency & Documentation To what extent are the model outputs consistent with the entity’s values and principles to foster public trust and equity? Organizations can document the following:

170