Web Scraping Yahoo! Finance using Python

A detailed guide for web scraping https://finance.yahoo.com/ using Requests, BeautifulSoup, Selenium, HTML tags & embedded JSON data.

Introduction

What is Web scraping?

Web scraping is the process of extracting and parsing data from websites in an automated fashion using a computer program. It's a useful technique for creating datasets for research and learning.

Objective

The main objective of this tutorial is to showcase different web scraping methods which can be applied to any web page.
This is for educational purposes only. Please read the Terms & Conditions carefully for any website to see whether you can legally use the data.

In this project, we will perform web scraping using the following 3 techniques based on the problem statement.

• use Requests, BeautifulSoup and HTML tags to extract web page
• use Selenium to scrape data from dynamically loading websites
• use embedded JSON data to scrape website

The problem statement

1. Web Scraping Stock Market News (url : https://finance.yahoo.com/topic/stock-market-news/)
   
   This web page shows the latest news related to stock market, we will try to extract data from this web page and store it in a CSV (comma-separated values) file. The file layout would be as mentioned below.
   
   

2. Web Scraping Cryptocurrencies (url : https://finance.yahoo.com/cryptocurrencies)
   
   This Yahoo! finance web page shows list of trending Cryptocurrencies in tabular format, we will perform the web scraping to retrieve first 10 columns for top 100 Cryptocurrencies in following CSV format.
   
   

3. Web Scraping Market Events Calendar (url : https://finance.yahoo.com/calendar)
   
   This page shows date-wise market events, user have the option to select the date and choose any one of the following market events Earnings, Stock Splits, Economic Events & IPO. Our aim is to create a script which can be run for any single date and market event which grabs the data and loads in CSV format as shown below.
   
   

Prerequisites

• Knowledge of Python
• Basic knowledge of HTML although it is not necessary

How to run the Code

You can execute the code using "Run" button on the top of this page and selecting "Run on Colab" or "Run Locally"




Setup and Tools

Run on Colab :
You will need to provide the Google login to run this notebook on Colab.

Run Locally : Download and install Anaconda framework, We will be using Jupyter Notebook for writing & executing code.

Version control

You can make changes and save your version of the notebook to Jovian by executing following cells.

1. Web Scraping Stock Market News

In this section we will learn a basic Python web scraping technique using Requests, BeautifulSoup and HTML tags. The objective here is to perform web scraping of Yahoo! finance Stock Market News

Let's kick-start with the first objective. Here's an outline of the steps we'll follow

1.1 Download & Parse web page using Requests and BeautifulSoup

1.2 Exploring and locating Elements

1.3 Extract & Compile the information into python list

1.4 Save the extracted information to a CSV file

1.1 Download & Parse webpage using Requests and BeautifulSoup

First step is to install requests & beautifulsoup4 Libraries using pip.

The libraries are installed and imported.

To download the page, we can use requests.get, which returns a response object. We can access the content of the web page using response.text

Also the response.ok & response.status_code can be used for error trapping & tracking.

Finally, we can use BeautifulSoup to parse the HTML data. This will return bs4.BeautifulSoup object.

Let's create a function to perform this step.

calling function get_page and analyzing the output.

You can access different properties of HTML web page from doc. Following example will display Title of the web page.

We can use the function get_page to download any web page and parse it using beautiful soup.

1.2 Exploring and locating Elements

Now its time to explore the elements to find the required data point from the web page. Web pages are written in a language called HTML (Hyper Text Markup Language). HTML is a fairly simple language comprised of tags (also called nodes or elements) e.g. <a href="https://finance.yahoo.com/" target="_blank">Go to Yahoo! Finance</a>. An HTML tag has three parts:

1. Name: (html, head, body, div, etc.) Indicates what the tag represents and how a browser should interpret the information inside it.
2. Attributes: (href, target, class, id, etc.) Properties of tag used by the browser to customize how a tag is displayed and decide what happens on user interactions.
3. Children: A tag can contain some text or other tags or both between the opening and closing segments, e.g., <div>Some content</div>.

Let's inspect the webpage source code by right-clicking and selecting the "Inspect" option. First, we need to identify the tag which represents the news listing.

In this case we can see the <div> tag having class name "Ov(h) Pend(44px) Pstart(25px)" is representing news listing. We can apply find_all method to grab this information

Total elements in the <div> tag list matching with the numbers of news displaying on the webpage, so we are heading towards the right direction.

Next step to inspect the individual <div> tag and try to find more information. I am using "Visual Studio Code", but you can use any tool as simple as notepad.

Luckily, most of the required data points are available in this <div>, so we can use find method to grab each item.

If any tag is not accessible directly, then you can use methods like findParent() or 'findChild() to point to the required tag.

Key Takeout from this exercise is to identify the optimal tag which will provide us required information. Mostly this is straight forward, but sometimes you will have to perform a little more research.

1.3 Extract & Compile the information into python list

We've identified all the required tags and information. Let's put this together in the functions.

sample run of the function get_news_tags

we will create one more function, to parse individual <div> tags and return the information in dictionary form

Testing the parse_news function for first <div> tag

We can use the get_news_tags & parse_news functions to pars news.

1.4 Save the extracted information to a CSV file

This is the last step of this section. We are going to use Python library pandas to save the data in CSV format. Install and then import the pandas Library.

Creating wrapper function which will call previously created helper functions.

The get_page function will download HTML page, then we can pass the result in get_news_tags to identify list of <div> tags for news.

After that we will use List Comprehension technique to parse each <div> tag using parse_news, the output will be in the form of lists of dictionaries

Finally, we will use DataFrame method to create pandas dataframe and use to_csv method to store required data in CSV format.

Scraping the news using scrape_yahoo_news function

The "stock-market-news.csv" should be available in File $\rightarrow$ Open Menu. You can download the file or directly open it on browser. Please verify the file content and compare it with the actual information available on the webpage.

You can also check the data by grabbing a few rows from the data frame returned by the scrape_yahoo_news function

Summary : Hopefully I was able to explain this simple but very powerful Python technique to scrape the Yahoo! finance market news. These steps can be used to scrape any web page. You just have to do a little research to identify the required <tags> and use relevant python methods to collect the data.

2. Web Scraping Cryptocurrencies

In phase One we were able to scrape the yahoo market news web page. However, if you've noticed, as we scroll down the webpage more news will appear at the bottom of the page. This is called dynamic page loading. The previous technique is a basic Python method useful to scrape static data, To scrape the dynamically loading data will use a different method called web scraping using Selenium. Let's move ahead with this topic. The goal of this section is to extract top listing Crypto currencies from Yahoo! finance.

Here's an outline of the steps we'll follow

2.1 Introduction of selenium

2.2 Downloads & Installation

2.3 Install & Import libraries

2.4 Create Web Driver

2.5 Exploring and locating Elements

2.6 Extract & Compile the information into a python list

2.7 Save the extracted information to a CSV file

2.1 Introduction of selenium

Selenium is an open-source web-based automation tool. Python language and other languages are used with Selenium for testing as well as web scraping. Here we will use Chrome browser, but you can try on any browser.

Why you should use Selenium?

• Clicking on buttons
• Filling forms
• Scrolling
• Taking a screen-shot
• Refreshing the page

You can find proper documentation on selenium here

The following methods will help to find elements in a webpage (these methods will return a list):

• find_elements_by_name
• find_elements_by_xpath
• find_elements_by_link_text
• find_elements_by_partial_link_text
• find_elements_by_tag_name
• find_elements_by_class_name
• find_elements_by_css_selector

In this tutorial we will use only find_elements_by_xpath and find_elements_by_tag_name You can find complete documentation of these methods here

2.2 Downloads & Installation

Unlike the previous section, here we'll have to do some prep work to implement this method. We will need to install Selenium & proper web browser driver

If you are using Google Colab platform then execute following code to perform Initial installation. This piece of code 'google.colab' in str(get_ipython()) is used to identify the Google Colab platform.

To run it on Locally you will need Webdriver for Chrome on your machine. You can download it from this link https://chromedriver.chromium.org/downloads and just copy the file in the folder where we will create the python file (No need of installation). But make sure that the driver‘s version matches the Chrome browser version installed on the local machine.

2.3 Install & Import libraries

Installation of the required libraries.

Once the Libraries installation is done, next step is to import all the required modules / libraries.

So all the necessary prep work is done. Let's move ahead to implement this method.

2.4 Create Web Driver

In this step first we will create the instance of Chrome WebDriver using webdriver.Chrome() method. and then the driver.get() method will navigate to a page given by the URL. In this case also there is slight variation based on platform. Also we have used options parameters for e.g. --headless option will load the driver in background.

Test run of get_driver

2.5 Exploring and locating Elements

This is almost similar step that we have done in phase 1. We will try to identify relevant information like <tags>, class , XPath etc from the web page. Right-click and select the "Inspect" to do further analysis.

As the webpage showing cryptocurrency information in the Table form. We can grab the table header by using tag <th>, we will use find_elements by TAG to get the table headers. These headers can be used as columns for a CSV file.

Creating a helper function to get first 10 columns from header, we have used List comprehension with conditions. You can also check out usage of enumerate method.

Next we find out number of rows available in a Page, you can see table rows are placed in <tr> tag, we can capture the XPath by selection <tr> tag the Right Click $\rightarrow$ Copy $\rightarrow$ Copy XPath.

So we get the XPath value as //*[@id="scr-res-table"]/div[1]/table/tbody/tr[1], Let's use this with find_element() & By.XPATH.

Above XPath points to first row, we can get rid of row number part from XPath and use it with find_elements to get hold of all the available rows. Let's implement this in a function.

Similarly, we can take the XPath for any column value.

This is the XPAth for a column //*[@id="scr-res-table"]/div[1]/table/tbody/tr[1]/td[2].

If you noticed the number after tr & td represents the row_number and column_number, we can check this with find_element() method

So we can change the row_number & column_number in XPath and loop it through row count and column count to get all the available column values. Let's generalize this and put it in a function. We will get the data for one row at a time and return column values in the form of a dictionary

The Yahoo! Finance web page shows only 25 Cryptocurrencies per page and user will have to click Next button to load next set of crypto currencies. This is called Pagination. This is the main reason we are implementing selenium methods to handle events like pagination. you can perform multiple events like clicking, scrolling , refreshing etc. on a webpage using selenium methods.

Now we will grab the XPath of Next button, find the element using find_element method, and after that we can perform click action using .click() method

Now I am trying to check the first row on the web page to verify if .click() really worked, and you will see first row has changed. Click action was successful.

In this section we have learned how to get required data points, and perform events on webpage.

2.6 Extract & Compile the information into python list

Let's put all the pieces in the puzzle, we will pass the integer total_crypto i.e. numbers of rows to be scraped (in this case 100 rows) in the function. Parse each row from the page and append the data in the List till the total parsed row count reach to total_crypto. In addition, we will perform Next button click if we are at the last row of the table.

Please Note : Here to identify the Next button element we have used WebDriverWait class instead of using find_element() method. In this technique we can pass some wait-time before grabbing the element. This type of implementation is done to avoid the StaleElementReferenceException.

Code Sample:

element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[@id="scr-res-table"]/div[2]/button[3]')))

2.7 Save the extracted information to a CSV file.

This is the last step of this section, we are creating a last function which will be the placeholder for all helper functions and at the and we will save the data in CSV format using pd.to_csv method.

Time to scrape some cryptos!!! , we will scrape top 100 cryptos in Yahoo! Finance webpage by calling scrape_yahoo_crypto

The "crypto-currencies.csv" should be available in File $\rightarrow$ Open Menu. You can download the file or directly open it on browser. Please verify the file content and compare it with the actual information available on the webpage.

You can also check the data by grabbing a few rows from the data frame returned by the scrape_yahoo_crypto function

Summary : Hope you've enjoyed this tutorial. Selenium enables us to perform multiple actions on the web browser, which is really very handy for scraping different types of data from any webpage.

3. Web Scraping Market Events Calendar

This is the final segment of the tutorial in this section, we will learn how to extract embedded JSON formatted data which can be easily converted to Python dictionary. Problem statement for section is to scrape date-wise market events from Yahoo! finance.

Here's an outline of the steps we'll follow

3.1 Install & Import libraries

3.2 Download & Parse web page

3.3 Get Embedded Json data

3.4 Locating Json Keys

3.5 Pagination & Compiling the information into a python list

3.6 Save the extracted information to a CSV file

3.1 Install & Import libraries

First step to install and import Python Libraries

3.2 Download & Parse web page

This is exactly the same step that we've performed to download webpage in section 1.1, Here we have used custom header in requests.get()

Most of the things are explained in section 1.1, creating the helper function.

3.3 Get Embedded Json data

In this step we will locate the Jason formatted data, Open the web page and do Right Click $\rightarrow$ View Page Source, If you scroll down to source page you will notice the Json formatted data. Apparently this information is <script> tag which contains the following text /* -- Data -- */.

We will use Regular expressions to get text inside <script> tag.

Further, the Json formatted string has the first key as context and it ends at 12 characters from the end

We can grab the Json string using Python slicing.

Using json.loads()method to convert Jason string into Python Dictionary

Creating a function using above information.

3.4 Locating Json Keys

So basically the Json text is multi level nested dictionaries, and some keys are used to store all the meta data displayed on the webpage. In this section we will identify the keys for the data we are trying to scrape.

We'll need some Json Formatter tool to navigate through multiple keys, I am using online tool https://jsonblob.com/. However, you can choose any tool.

We will write the Json text into my_json_file.json file, then grab the file content and paste it to the left panel of https://jsonblob.com/. The JSON Blob it will do nice formatting. We can easily navigate through each Keys and search any item.

Next step is to find the Required Key location. Let's search the company name 3D Systems Corporation displayed in the webpage in the JSON Blob formatter.

You can see the table data is stored in the rows key, and we can track down the parent keys as shown in the above screen, checkout the content of row key.

This sub-dictionary shows all the data displayed on the current page.

You can do more research and exploration to get different information from the web page.

Putting this in a function

3.5 Pagination & Compiling the information into python list

As we saw in the previous section on how to handle Pagination using selenium methods, here we'll learn a new technique for accessing multiple pages.

Most of the times webpage url gets changed at runtime depending on the user selection, e.g. In the below screen-shot, I selected the Earnings for 1-March-2022. You can notice how that information is passed in the URL.

Similarly, when i click next button, offset& size values gets changed in the url.

So we can figure out the pattern & structure of the url and how it affects page navigation.

In this case webpage url pattern is mentioned below:

• The following values are used for calendar event types
  event_types = ['splits','economic','ipo','earnings']
• Date passed in yyyy-mm-dd format
• Page number is controlled by offset value (for first page offset=0)
• Maximum number of rows in a page is assigned to size

Based on the above information, we can build the URL at runtime and download the page, then extract the information. This is how we handle pagination.

Putting all things together in a function. In this function we will pass event_type and date, then we will calculate the total rows for matching criteria using get_total_rows function. Maximum rows per page are constant (i.e. 100), so we can build iterating summation logic to calculate the total number of pages involved for the current criteria and extract each page data in the loop.

creating one more variant of same function for the date range

3.6 Save the extracted information to a CSV file.

In this last section, we will save the data to csv format using pd.DataFrame() & to_csv() and call everything in a single placeholder function.

creating one more variant of same function for the date range

calling the final function scrape_yahoo_calendar_daterange

calling the final function scrape_yahoo_calendar

Total 4 csv files "event_type_yyyy-mm-dd.csv" should be available in File $\rightarrow$ Open Menu. You can download the file or directly open it in a browser. Please verify the file content and compare it with the actual information available on the webpage.

Summary : This is a very useful technique which can be easily replicable. Without writing any customized code, we were able to extract the data from multiple types of web pages just by changing one variable (in this case event_type).

References

References to some useful links.

• https://htmldog.com/guides/html/
• https://selenium-python.readthedocs.io/index.html
• https://stackoverflow.com/questions/27003423/staleelementreferenceexception-on-python-selenium
• https://www.w3schools.com/js/js_json_intro.asp
• https://hhsm95.dev/blog/the-importance-of-using-user-agent-to-scraping-data/

Future Work

Ideas for future work

• Automate this process using AWS Lambda to download daily market calendar, crypto-currencies & market news in CSV format.
• Move the old files to an Archive folder append date-stamp to the file if required, also delete the Archived files older than 2 weeks.
• Process the raw data extracted from third technique using different methods of pandas

Conclusion

In this tutorial we implement the following web scraping techniques.

• Use Requests, BeautifulSoup and HTML tags to extract web page.
• Use Selenium to scrape data from dynamically loading websites.
• Use embedded JSON data to scrape website.

I hope I was able to teach you these webscraping methods and I hope you can use this knowledge to scrape any website.

If you have any questions, feedback feel free to post a comment or contact me on LinkedIn. Thank you for reading and if you liked this post, please consider following me. Until next time… Happy coding !!

Don’t forget to give your 👏 !