Cyber & IT Supervisory Forum - Additional Resources

Did you establish mechanisms that facilitate the AI system’s auditability (e.g., traceability of the development process, the sourcing of training data and the logging of the AI system’s processes, outcomes, positive and negative impact)? Did you ensure that the AI system can be audited by independent third parties? Did you establish a process for third parties (e.g., suppliers, end-users, subjects, distributors/vendors or workers) to report potential vulnerabilities, risks or biases in the AI system? GAO-21-519SP - Artificial Intelligence: An Accountability Framework for Federal Agencies & Other Entities. Artificial Intelligence Ethics Framework for the Intelligence Community. AI Transparency Resources AI Incident Database. 2022. AIAAIC Repository. 2022. Netflix. Chaos Monkey. IBM. “IBM's Principles of Chaos Engineering.” IBM, n.d. Suchi Saria and Adarsh Subbaswamy. "Tutorial: Safe and Reliable Machine Learning." arXiv preprint, submitted April 15, 2019. Daniel Kang, Deepti Raghavan, Peter Bailis, and Matei Zaharia. "Model assertions for monitoring and improving ML models." Proceedings of Machine Learning and Systems 2 (2020): 481-496. Larysa Visengeriyeva, et al. “Awesome MLOps.“ GitHub. McGregor, S., Paeth, K., & Lam, K.T. (2022). Indexing AI Risks with Incidents, Issues, and Variants. ArXiv, abs/2211.10384. References

130