Automating a price comparison tool using Python

In the retail world, a competitor price analysis is an important activity that Marketing departments regularly do as an effort to get the insgihts needed for setting their own product prices. Though knowking the competitor's product prices are not the only information required for setting a product price in retail, it is certainly important to be aware of it for reference.

There are multiple approaches for finding competitor's prices that varies in a wide range of time, effort, price, and accuracy of the results.

Some retailers ask their employees to research the competitor's prices by their own means which could include visiting stores, calling to Customer Service and/or copying manually data from the competitor's website. The list of prices that each employee gets is sent to another person who is accountable for compiling multiple files and providing an overall report.

Copying data manually from a website is a boring, tiring, time consuming, and prone to human error task.

There is a better approach for gathering massive information from a website that is called Web Scraping which is an automated technique for extracting required data from websites through software.

In this notebook, I'll walk you through the process of using Python libraries such as Requests, Selenium and Beautiful Soup for grabbing product prices by iterating a user-defined list of product categories from Farmacia San Pablo website and then fetching the price from a competitor website named Farmacias del Ahorro

The overall steps I'll follow are:

1. Understanding the structure of Farmacia San Pablo and Farmacias del Ahorro websites
2. Install Python libraries and modules
3. Download and parse Farmacia San Pablo webpage using Selenium and Beautiful Soup
4. Extract product details from Farmacia San Pablo
5. Lookup and extract regular and discount price from Farmacias del Ahorro
6. Combine extracted data from Farmacia San Pablo and Farmacias del Ahorro into a dictionary
7. Compiling all product details into a Pandas dataframe and creating a CSV file
8. Integrating all in a single main function
9. Execution of main function

By the end of the project, is expected to create a csv file with the following information:

product, description, url, price, upc, url website2, regular price website2, discount price website2
Biretix Micropeel Tratamiento Exfoliante,50 ML Tubo,https://www.farmaciasanpablo.com.mx/dermocosmeticos/dermocosmeticos/geles-o-lociones-de-limpieza/biretix-micropeel-tratamiento-exfoliante/p/000000000005130301,385.0,8470001729057,377.0,358.15

How to run the code

Press Run button above.
Also refer to Execution of main function step at the end of the notebook

Understanding the structure of Farmacia San Pablo and Farmacias del Ahorro websites

I'm going to gather product prices from two leading companies that are in the business of pharmacy retail located in Mexico named 'Farmacia San Pablo' also known as FSP and 'Farmacias del Ahorro'; FA for short.