Evaluation of Diagnostic Models

Welcome to the second assignment of course 1. In this assignment, we will be working with the results of the X-ray classification model we developed in the previous assignment. In order to make the data processing a bit more manageable, we will be working with a subset of our training, and validation datasets. We will also use our manually labeled test dataset of 420 X-rays.

As a reminder, our dataset contains X-rays from 14 different conditions diagnosable from an X-ray. We'll evaluate our performance on each of these classes using the classification metrics we learned in lecture.

Outline

Click on these links to jump to a particular section of this assignment!

• 1. Packages
• 2. Overview
• 3. Metrics
  • 3.1 True Positives, False Positives, True Negatives, and False Negatives
  • 3.2 Accuracy
  • 3.3 Prevalence
  • 3.4 Sensitivity and Specificity
  • 3.5 PPV and NPV
  • 3.6 ROC Curve
• 4. Confidence Intervals
• 5. Precision-Recall Curve
• 6. F1 Score
• 7. Calibration

By the end of this assignment you will learn about:

1. Accuracy
2. Prevalence
3. Specificity & Sensitivity
4. PPV and NPV
5. ROC curve and AUCROC (c-statistic)
6. Confidence Intervals

1. Packages

In this assignment, we'll make use of the following packages:

• numpy is a popular library for scientific computing
• matplotlib is a plotting library compatible with numpy
• pandas is what we'll use to manipulate our data
• sklearn will be used to measure the performance of our model

Run the next cell to import all the necessary packages as well as custom util functions.