Cyber & IT Supervisory Forum - Additional Resources

Given the purpose of this AI, what is an appropriate interval for checking whether it is still accurate, unbiased, explainable, etc.? What are the checks for this model? How has the entity identified and mitigated potential impacts of bias in the data, including inequitable or discriminatory outcomes? To what extent has the entity identified and mitigated potential bias— statistical, contextual, and historical—in the data? Were adversarial machine learning approaches considered or used for measuring bias (e.g.: prompt engineering, adversarial models) GAO-21-519SP - Artificial Intelligence: An Accountability Framework for Federal Agencies & Other Entities. Artificial Intelligence Ethics Framework for the Intelligence Community. WEF Companion to the Model AI Governance Framework- WEF - Companion to the Model AI Governance Framework, 2020. Datasheets for Datasets. Ali Hasan, Shea Brown, Jovana Davidovic, Benjamin Lange, and Mitt Regan. “Algorithmic Bias and Risk Assessments: Lessons from Practice.” Digital Society 1 (2022). Richard N. Landers and Tara S. Behrend. “Auditing the AI Auditors: A Framework for Evaluating Fairness and Bias in High Stakes AI Predictive Models.” American Psychologist 78, no. 1 (2023): 36–49. Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. “A Survey on Bias and Fairness in Machine Learning.” ACM Computing Surveys 54, no. 6 (July 2021): 1–35. Michele Loi and Christoph Heitz. “Is Calibration a Fairness Requirement?” FAccT '22: 2022 ACM Conference on Fairness, Accountability, and Transparency, June 2022, 2026–34. Shea Brown, Ryan Carrier, Merve Hickok, and Adam Leon Smith. “Bias Mitigation in Data Sets.” SocArXiv, July 8, 2021. URL Reva Schwartz, Apostol Vassilev, Kristen Greene, Lori Perine, Andrew Burt, and Patrick Hall. "NIST Special Publication 1270 Towards a Standard for Identifying and Managing Bias in Artificial Intelligence." National Institute of Standards and Technology (NIST), 2022. References AI Transparency Resources

149