Exploratory Data Analysis using Python - A Case Study

Analyzing responses from the Stack Overflow Annual Developer Survey 2020

Part 9 of "Data Analysis with Python: Zero to Pandas"

This tutorial is the ninth in a series on the introduction to programming and data analysis using the Python language. These tutorials take a practical coding-based approach, and the best way to learn the material is to execute the code and experiment with the examples. Check out the full series here:

  1. First Steps with Python and Jupyter
  2. A Quick Tour of Variables and Data Types
  3. Branching using Conditional Statements and Loops
  4. Writing Reusable Code Using Functions
  5. Reading from and Writing to Files
  6. Numerical Computing with Python and Numpy
  7. Analyzing Tabular Data using Pandas
  8. Data Visulation using Matplotlib & Seaborn
  9. Exploratory Data Analysis - A Case Study

How to run the code

This tutorial hosted on Jovian.ml, a platform for sharing data science projects online. You can "run" this tutorial and experiment with the code examples in a couple of ways: using free online resources (recommended) or on your own computer.

This tutorial is a Jupyter notebook - a document made of "cells", which can contain explanations in text or code written in Python. Code cells can be executed and their outputs e.g. numbers, messages, graphs, tables, files etc. can be viewed within the notebook, which makes it a really powerful platform for experimentation and analysis. Don't afraid to experiment with the code & break things - you'll learn a lot by encoutering and fixing errors. You can use the "Kernel > Restart & Clear Output" menu option to clear all outputs and start again from the top of the notebook.

Option 1: Running using free online resources (1-click, recommended)

The easiest way to start executing this notebook is to click the "Run" button at the top of this page, and select "Run on Binder". This will run the notebook on mybinder.org, a free online service for running Jupyter notebooks. You can also select "Run on Colab" or "Run on Kaggle", but you'll need to create an account on Google Colab or Kaggle to use these platforms.

Option 2: Running on your computer locally

You'll need to install Python and download this notebook on your computer to run in locally. We recommend using the Conda distribution of Python. Here's what you need to do to get started:

  1. Install Conda by following these instructions. Make sure to add Conda binaries to your system PATH to be able to run the conda command line tool from your Mac/Linux terminal or Windows command prompt.

  2. Create and activate a Conda virtual environment called zerotopandas which you can use for this tutorial series:

conda create -n zerotopandas -y python=3.8 
conda activate zerotopandas

You'll need to create the environment only once, but you'll have to activate it every time want to run the notebook. When the environment is activated, you should be able to see a prefix (zerotopandas) within your terminal or command prompt.

  1. Install the required Python libraries within the environment by the running the following command on your terminal or command prompt:
pip install jovian jupyter numpy pandas matplotlib seaborn opendatasets --upgrade
  1. Download the notebook for this tutorial using the jovian clone command:
jovian clone aakashns/python-eda-stackoverflow-survey

The notebook is downloaded to the directory python-eda-stackoverflow-survey.

  1. Enter the project directory and start the Jupyter notebook:
cd python-eda-stackoverflow-survey
jupyter notebook
  1. You can now access Jupyter's web interface by clicking the link that shows up on the terminal or by visiting http://localhost:8888 on your browser. Click on the notebook python-eda-stackoverflow-survey.ipynb to open it and run the code. If you want to type out the code yourself, you can also create a new notebook using the "New" button.

Introduction

We'll use the StackOverflow developer survey dataset for our analysis. This is an annual survey conducted by StackOverflow, and you can find the raw data & results here: https://insights.stackoverflow.com/survey.

There are several options for getting the dataset into Jupyter:

  • Download the CSV manually and upload it via Jupyter's GUI
  • Use the urlretrieve function from the urllib.request to download CSV files from a raw URL directly
  • Use a helper library e.g. opendatasets, which contains a collection of curated datasets and provides a function for directly download.

We'll use the opendatasets helper library to download the files.

import opendatasets as od
od.download('stackoverflow-developer-survey-2020')
Using downloaded and verified file: ./stackoverflow-developer-survey-2020/survey_results_public.csv Using downloaded and verified file: ./stackoverflow-developer-survey-2020/survey_results_schema.csv Using downloaded and verified file: ./stackoverflow-developer-survey-2020/README.txt