Cyber & IT Supervisory Forum - Additional Resources

Collect uses cases from the operational environment for system testing and monitoring activities in accordance with organizational policies and regulatory or disciplinary requirements (e.g., informed consent, institutional review board approval, human research protections),

Organizations can document the following: Transparency & Documentation

To what extent is the output of each component appropriate for the operational context? What justifications, if any, has the entity provided for the assumptions, boundaries, and limitations of the AI system? How will the appropriate performance metrics, such as accuracy, of the AI be monitored after the AI is deployed? As time passes and conditions change, is the training data still representative of the operational environment? GAO-21-519SP - Artificial Intelligence: An Accountability Framework for Federal Agencies & Other Entities. Artificial Intelligence Ethics Framework for the Intelligence Community. AI Transparency Resources Luca Piano, Fabio Garcea, Valentina Gatteschi, Fabrizio Lamberti, and Lia Morra. “Detecting Drift in Deep Learning: A Methodology Primer.” IT Professional 24, no. 5 (2022): 53–60. Larysa Visengeriyeva, et al. “Awesome MLOps.“ GitHub. References

123