Web Scraping Job Postings from Indeed.com

The job board Indeed.com has become a leading worldwide employment website since its founding in 2004. One of the features that makes it so prominent is that it aggregates job postings from multiple sources such as other websites, job boards, staffing companies, and more. In addition, Indeed allows for job searches using multiple parameters all at once to give focused results for a job seeker.

This web scraping project details how to extract the basic information for job postings spanning multiple pages in an Indeed search using various parameters.

We make use of three main Python libraries to accomplish the web scraping project:

• Requests (This is used to get the HTML source code for a webpage)
• Beautiful Soup (This is used to parse the HTML documents)
• Pandas (This is used to collect the data into a readable and presentable format)

This notebook is divided into the following sections:

Section 1: Scraping the first webpage of an Indeed search

Section 2: Scraping multiple pages of an Indeed search

Section 3: Scraping using multiple parameters

Section 4: Generating CSV files of Indeed searches

First we import the relevant libraries