Cyber & IT Supervisory Forum - Additional Resources

Measure 2 AI systems are evaluated for trustworthy characteristics. Measure 2.1 Test sets, metrics, and details about the tools used during test, evaluation, validation, and verification (TEVV) are documented. About Documenting measurement approaches, test sets, metrics, processes and materials used, and associated details builds foundation upon which to build a valid, reliable measurement process. Documentation enables repeatability and consistency, and can enhance AI risk management decisions. Suggested Actions Leverage existing industry best practices for transparency and documentation of all possible aspects of measurements. Examples include: data sheet for data sets, model cards, [commenters provided examples]. Regularly assess the effectiveness of tools used to document measurement approaches, test sets, metrics, processes and materials used. Update the tools as needed. Given the purpose of this AI, what is an appropriate interval for checking whether it is still accurate, unbiased, explainable, etc.? What are the checks for this model? To what extent has the entity documented the AI system’s development, testing methodology, metrics, and performance outcomes? GAO-21-519SP - Artificial Intelligence: An Accountability Framework for Federal Agencies & Other Entities. Artificial Intelligence Ethics Framework for the Intelligence Community. AI Transparency Resources Organizations can document the following: Transparency & Documentation

113