Cyber & IT Supervisory Forum - Additional Resources

Organizations can document the following: Transparency & Documentation

What metrics has the entity developed to measure performance of the AI system? To what extent do the metrics provide accurate and useful measure of performance? What corrective actions has the entity taken to enhance the quality, accuracy, reliability, and representativeness of the data? How will the accuracy or appropriate performance metrics be assessed? What is the justification for the metrics selected? GAO-21-519SP - Artificial Intelligence: An Accountability Framework for Federal Agencies & Other Entities. URL Artificial Intelligence Ethics Framework for the Intelligence Community. AI Transparency Resources ACM Technology Policy Council. “Statement on Principles for Responsible Algorithmic Systems.” Association for Computing Machinery (ACM), October 26, 2022. Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. Springer-Verlag, 2009. Harini Suresh and John Guttag. “A Framework for Understanding Sources of Harm Throughout the Machine Learning Life Cycle.” Equity and Access in Algorithms, Mechanisms, and Optimization, October 2021. Christopher M. Bishop. Pattern Recognition and Machine Learning. New York: Springer, 2006. Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, W. Duncan Wadsworth, and Hanna Wallach. “Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs.” Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, July 2021, 368–78. References

109