Jovian
Sign In
Learn practical skills, build real-world projects, and advance your career
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All" 
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session
/kaggle/input/house-prices-advanced-regression-techniques/housetrain.csv /kaggle/input/titanicdataset-traincsv/train.csv

Missing Values- Feature Engineering

Lifecycle of a Data Science Projects

Data Collection Statergy---> from company side, 3rd party APi's, Surveys, Surveys

Feature Engineering--->Handling Missing Values

Why are their Missing values??

  • People hesitate to put down the information like in Depression Surveys
  • Survey informations are not that valid
  • Men hesitate to put their salary
  • Women generally doesn't disclose their age
  • People may have died----NAN

Data Science Projects--->Dataset should be collected from multiple sources

What are the different types of Missing Data?

  1. MCAR (Missing Completely at Random)
  2. MNAR (Missing data not at Random) : Systematic missing values
  3. Missing At Random(MAR)

1. MCAR (Missing Completely at Random)

Missing Completely at Random, MCAR: A variable is missing completely at random (MCAR) if the probability of being missing is the same for all the observations. When data is MCAR, there is absolutely no relationship between the data missing and any other values, observed or missing, within the dataset. In other words, those missing data points are a random subset of the data. There is nothing systematic going on that makes some data more likely to be missing than other.

#titanic dataset
df=pd.read_csv('/kaggle/input/titanicdataset-traincsv/train.csv')