Scraping Yahoo Finance Historical Stock Data using Selenium

Programmer’s Picnic · Learn with Champak Roy

Why Requests Fails for Yahoo Finance

Yahoo Finance loads its historical price table using JavaScript. When we use the requests library, Python only downloads the initial HTML — it does not execute JavaScript.

As a result, the historical data table is missing.

📌 Rule of Thumb:
If data appears only after the page loads → Selenium is required.

Why Selenium Works

Selenium opens a real browser (Chrome), executes JavaScript, and allows us to interact with dynamically loaded content.

Working Python Code (Yahoo Finance History)

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from time import sleep
from stockslist import stocks

# ---------- INPUT ----------
stockname = input("Enter stock name: ").upper()
stockurl = stocks[stockname]

# ---------- DRIVER ----------
driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install())
)

driver.get(stockurl)

# Wait for JavaScript-loaded content
sleep(5)

# ---------- SCRAPE TABLE ----------
rows = driver.find_elements(
    By.XPATH, '//tr[@data-test="historical-prices"]'
)

print("\\nDate | Open | High | Low | Close | Volume")
print("-" * 60)

for row in rows:
    cols = row.find_elements(By.TAG_NAME, "td")
    if len(cols) == 7:
        data = [c.text for c in cols]
        print(" | ".join(data))

driver.quit()

What This Program Does

  1. Takes a stock name as input
  2. Opens Yahoo Finance historical page
  3. Waits for JavaScript to load data
  4. Extracts historical prices row by row
  5. Prints clean tabular output
🎓 This is a real industry scraping pattern used in finance, analytics, and ML pipelines.

Next Upgrades