Advanced Data Analysis Techniques

External Resources

• Google Colab Tricks: https://towardsdatascience.com/10-tips-for-a-better-google-colab-experience-33f8fe721b82
• Sources for finding datasets online-
  • Kaggle datasets - https://www.kaggle.com/datasets
  • World Bank Open Data - https://data.worldbank.org/
  • Yahoo Finance - https://finance.yahoo.com/
  • Google Dataset Search - https://datasetsearch.research.google.com/
  • UCI Machine Learning Repository - https://archive.ics.uci.edu/ml/datasets.php
  • FastAI datasets - https://course.fast.ai/datasets
• Steps to find your Kaggle credentials - https://github.com/jovianml/opendatasets#kaggle-credentials
• Comparison of the feather and parquet formats - https://ursalabs.org/blog/2020-feather-v2/
• IO Tools documentation - https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html
• More about working with categorical data in Pandas - https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html
• More string methods supported by Pandas data frames and series - https://pandas.pydata.org/docs/user_guide/text.html#string-methods
• More date methods supported by Pandas series - https://pandas.pydata.org/pandas-docs/stable/user_guide/basics.html#basics-dt-accessors
• More about map and apply - https://towardsdatascience.com/introduction-to-pandas-apply-applymap-and-map-5d3e044e93ff
• More about dataframe concatenation - https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.html#concatenating-objects
• Working with categorical data in Pandas - https://jovian.ai/himani007/categorical-data-with-pandas
• Working with large datasets in Pandas - https://jovian.ai/himani007/pandas1-large-datasets
• Python for Data Analysis - https://www.amazon.com/Python-Data-Analysis-Wrangling-IPython-ebook/dp/B075X4LT6K
• Pandas API reference (overview of all public pandas objects, functions and methods) - https://pandas.pydata.org/pandas-docs/stable/reference/index.html
• Merging Pandas dataframes - https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.html
• Advanced Pandas tutorial notebooks - https://www.kaggle.com/residentmario/welcome-to-advanced-pandas
• How to load CSV files 10x faster and use 10x less memory - https://towardsdatascience.com/%EF%B8%8F-load-the-same-csv-file-10x-times-faster-and-with-10x-less-memory-%EF%B8%8F-e93b485086c7
• More information about US-accidents dataset - https://smoosavi.org/datasets/us_accidents
• More encoding techniques with category_encoder - https://contrib.scikit-learn.org/category_encoders/
• Article on Categorical data with big data using PySpark - https://towardsdatascience.com/feature-encoding-with-spark-2-3-0-part-1-9ede45562740
• Article on types of categorical data encoding - https://www.analyticsvidhya.com/blog/2020/08/types-of-categorical-data-encoding/
• Article on multiple ways to load Kaggle datasets into Colab - https://medium.com/analytics-vidhya/how-to-fetch-kaggle-datasets-into-google-colab-ea682569851a
• Python strftime cheatsheet - https://strftime.org/
• More about parallel computing - https://www.techopedia.com/definition/8777/parallel-computing
• Tutorial on Dask - https://www.machinelearningplus.com/python/dask-tutorial/
• Article on how Dask uses disk space to reduce memory - https://medium.com/analytics-vidhya/a-deep-dive-into-dask-dataframes-7455d66a5bc5
• Dask documentation - https://docs.dask.org/en/latest/dataframe.html
• Datatable documentation - https://datatable.readthedocs.io/en/latest/start/index-start.html#
• RAPIDS documentation - https://docs.rapids.ai/
• PySpark documentation - https://spark.apache.org/docs/latest/api/python/index.html
• More about feather - https://arrow.apache.org/docs/python/feather.html
• Article about top 5 problems with big data and how to solve them - https://www.piesync.com/blog/top-5-problems-with-big-data-and-how-to-solve-them/
• Tutorial on how to speed up Pandas projects - https://realpython.com/fast-flexible-pandas/#this-tutorial
• User guide to scaling to larger datasets - https://pandas.pydata.org/pandas-docs/stable/user_guide/scale.html
• Article on Chunking - https://towardsai.net/p/data-science/efficient-pandas-using-chunksize-for-large-data-sets-c66bf3037f93
• More about Chunking - https://pythonspeed.com/articles/chunking-pandas/
• Reducing memory usage cheatsheet - https://attachments.convertkitcdnm.com/147732/2a30f1c4-8afa-4414-aed1-4d4994504ec1/reducememory.pdf
• Fast subsets of large datasets - https://pythonspeed.com/articles/indexing-pandas-sqlite/
• Lossy compression - https://pythonspeed.com/articles/pandas-reduce-memory-lossy/
• Article on faster data loading for Pandas - https://blog.esciencecenter.nl/irregular-data-in-pandas-using-c-88ce311cb9ef
• Apply method usage in Pandas: https://blog.jovian.ai/how-to-use-apply-method-in-pandas-a369ddd351a3

Common Questions

Q1: What is the size limit for Google Colab?

A: From 500MB to 1GB is the limit for both Colab and Binder.

Q2: Do we need to import Jovian to use opendatasets?

A: No, you don't have to import Jovian to use opendatasets.

Q3: Where can I get my Kaggle token from?

A: Go to your Kaggle account → Click on your profile at the top right → Choose account → Scroll down to API → Click on Create New API Token → that downloads a JSON file, use the credentials in it.

Reference - https://github.com/jovianml/opendatasets#kaggle-credentials

Q4: Is Google Colab secure or can the projects be accessed publicly by anyone?

A: Colab link is by default private and secure unless you make the link public.

Q5: What is walltime?

A: Time taken from command being issued to result being received. Wall time is total time of the operation.

Q6: Is it better to use Colab while working with large datasets instead of Binder and Kaggle?

A: Colab and Binder are better than Kaggle as of now because Jovian is not strongly integrated with Kaggle and cannot bring your notebooks back to Jovian. If you want to bring your notebooks back to Jovian, you have to use Binder or Colab.

Q7: Can I install geopandas on Colab?

A: Yes, you can install geopandas on Colab.

Q8: When to use join()? When to use merge()?

A: Always use merge() as join() can only work on indices.

Q9: What does inplace=True do?

A: inplace=True tells Pandas to modify the existing dataframe instead of creating a copy.

Q10: Do we need to use the Jovian account to run on Colab?

A: No, you can connect a different email account.