Learn practical skills, build real-world projects, and advance your career
Created 4 years ago
AI for Medicine Course 1 Week 1 lecture exercises
Patient Overlap and Data Leakage
Patient overlap in medical data is a part of a more general problem in machine learning called data leakage. To identify patient overlap in this week's graded assignment, you'll check to see if a patient's ID appears in both the training set and the test set. You should also verify that you don't have patient overlap in the training and validation sets, which is what you'll do here.
Below is a simple example showing how you can check for and remove patient overlap in your training and validations sets.
# Import necessary packages
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import os
import seaborn as sns
sns.set()Read in the data from a csv file
First, you'll read in your training and validation datasets from csv files. Run the next two cells to read these csvs into pandas dataframes.
# Read csv file containing training data
train_df = pd.read_csv("nih/train-small.csv")
# Print first 5 rows
print(f'There are {train_df.shape[0]} rows and {train_df.shape[1]} columns in the training dataframe')
train_df.head()There are 1000 rows and 16 columns in the training dataframe