CHAI Responsible AI Content documentation

This documentation includes use case folders with consensus-defined Testing and Evaluation Frameworks. When completing the CHAI Applied Model Card, review use case-specific Testing and Evaluation Frameworks for additional CHAI consensus-defined guidance.

A Testing and Evaluation (T&E) Framework includes a consensus-defined set of methods, metrics, measures, and/or benchmarks for developers and implementers to more concretely evaluate the responsible use of health AI solutions for a given use case. Content within a T&E Framework varies based on use case and type.

By releasing this content under an open-source license, we:

1. Welcome feedback and new evidence/examples for or against the use of a particular method or metric for a given use case. We have two methods for individuals to provide feedback once information is posted to the public repository.

   • For those who use GitHub, we have created a GitHub Issue form.

   • For those who would rather not use GitHub, we have created a Microsoft Form, which will automatically generate a GitHub Issue in the repository.

   • Issues raised will be prioritized, changes determined, committed, and reflected through version control (see Governance Structure and Contributing to CHAI).

2. Aim to stay relevant and up to date in a rapidly changing and growing field. We recognize that methods, metrics, and evidence in some areas are emerging, while in others they may be more established.

3. Want to prioritize getting information to our community rapidly and in a way that is useful. Waiting for documents to be “ideal” in a space where there are still uncertainties and unknowns, often delays dissemination.

4. Want to support the development of an open-source community that includes CHAI members and other domain experts who can begin testing these methods and metrics, providing examples, and filling any gaps in information.

5. Important Notes:

   • The content you see below will have blanks. This may be because it was outside of the scope of a specific workgroup focus, because the domain was added after content development, or because information still needs to be added. We welcome these additions from our community and other domain experts.

   • Content that is included was agreed upon by workgroup members and leads (see Governance Structure and Contributing to CHAI).

Contents:

• Community
  • Contributing to CHAI
  • Governance Structure
• Note Summarization - Patient Discharge Summary
  • Introduction
  • Testing and Evaluation (T&E) Framework
  • Datasets
  • Contributors
• Sepsis Risk Prediction
  • Introduction
  • Testing and Evaluation (T&E) Framework
  • Contributors
• General Health Advice Chatbot
  • Introduction
  • Testing and Evaluation (T&E) Framework
  • Contributors
  • Definitions
• Clinical Decision Support
  • Introduction
  • Testing and Evaluation (T&E) Framework
  • Contributors
  • Definitions
• Electronic Health Record (EHR) Information Retrieval
  • Introduction
  • Testing and Evaluation (T&E) Framework
  • Contributors
  • Definitions
• Prior Authorization AI-Supported Criteria Matching
  • Introduction
  • Testing and Evaluation (T&E) Framework
  • Contributors
  • Definitions