Cyber & IT Supervisory Forum - Additional Resources

Verify documentation policies for AI systems are standardized across the organization and remain current. Establish policies for a model documentation inventory system and regularly review its completeness, usability, and efficacy. Establish mechanisms to regularly review the efficacy of risk management processes. Identify AI actors responsible for evaluating efficacy of risk management processes and approaches, and for course-correction based on results. Establish policies and processes regarding public disclosure of the use of AI and risk management material such as impact assessments, audits, model documentation and validation and testing results. Document and review the use and efficacy of different types of transparency tools and follow industry standards at the time a model is in use. To what extent has the entity clarified the roles, responsibilities, and delegated authorities to relevant stakeholders? What are the roles, responsibilities, and delegation of authorities of personnel involved in the design, development, deployment, assessment and monitoring of the AI system? How will the appropriate performance metrics, such as accuracy, of the AI be monitored after the AI is deployed? How much distributional shift or model drift from baseline performance is acceptable?

Transparency & Documentation Organizations can document the following:

AI Transparency Resources

GAO-21-519SP: AI Accountability Framework for Federal Agencies & Other Entities. URL Intel.gov: AI Ethics Framework for Intelligence Community - 2020.

10