Cyber & IT Supervisory Forum - Additional Resources

Measure 3.3 Feedback processes for end users and impacted communities to report problems and appeal system outcomes are established and integrated into AI system evaluation metrics. About Assessing impact is a two-way effort. Many AI system outcomes and impacts may not be visible or recognizable to AI actors across the development and deployment dimensions of the AI lifecycle, and may require direct feedback about system outcomes from the perspective of end users and impacted groups. Feedback can be collected indirectly, via systems that are mechanized to collect errors and other feedback from end users and operators. Metrics and insights developed in this sub-category feed into Manage 4.1 and 4.2. Suggested Actions Measure efficacy of end user and operator error reporting processes. Categorize and analyze type and rate of end user appeal requests and results. Measure feedback activity participation rates and awareness of feedback activity availability. Utilize feedback to analyze measurement approaches and determine subsequent courses of action. Evaluate measurement approaches to determine efficacy for enhancing organizational understanding of real-world impacts. Analyze end user and community feedback in close collaboration with domain experts.

Organizations can document the following: Transparency & Documentation

To what extent can users or parties affected by the outputs of the AI system test the AI system and provide feedback? Did your organization address usability problems and test whether user interfaces served their intended purposes?

161