How to Scrape Indeed Jobs Data in 2025: A Step-by-Step Guide
Job data is crucial in today’s fast-paced job market. Whether you're an HR professional tracking hiring trends, a business analyzing job demands, or a job seeker looking for opportunities, having access to up-to-date job listings is a game-changer. Indeed, one of the largest job portals, offers a wealth of information — but manually sifting through job postings? That's time-consuming and inefficient.
Enter web scraping. It’s a way to automate data extraction from websites like Indeed, making it easy to track job openings, salary ranges, company trends, and more. And with API, scraping job data has never been easier or more reliable.
Why Scraping Indeed Job Listings Matters
Scraping Indeed allows businesses and professionals to get ahead in a competitive landscape. By automating data collection, you get a comprehensive view of job market trends, industry shifts, and the most in-demand roles. Forget the slow grind of manually searching for data — scraping offers speed, scale, and consistency.
For HR professionals, data scraping is the secret weapon to making better decisions about talent acquisition.
Mastering the API
API is built to handle complex scraping tasks without breaking a sweat. It can bypass anti-bot measures, ensuring you get uninterrupted, accurate data every time. Whether you need job titles, company names, or detailed descriptions, this tool simplifies the entire process.
Let’s walk through how to extract job data from Indeed, focusing on key details like job titles, descriptions, company names, and more.
Project Configuration
Ready to dive in? Here’s how to set things up.
Install Python 3.8 or Higher:
Make sure Python 3.8 or newer is installed on your machine. This guide works with Python 3.8+.
Build a Virtual Environment:
This is where you can store your project dependencies without cluttering your global Python setup. Here’s how to create one:
python -m venv indeed_env # For Windows
python3 -m venv indeed_env # For Mac/Linux
Turn On the Virtual Environment:
After creating the virtual environment, activate it:
.\indeed_env\Scripts\Activate # For Windows
source indeed_env/bin/activate # For Mac/Linux
Install Required Libraries:
You’ll need the requests library for HTTP requests and pandas for data manipulation. Install them with:
pip install requests pandas
Your environment is now ready to start scraping job data from Indeed.
How the API Works
The API simplifies scraping with minimal effort. Here’s a basic example:
import requests
payload = {
"source": "universal",
"url": "https://www.indeed.com"
}
response = requests.post(
url="https://api.example.com/v1/queries",
json=payload,
auth=("username", "password"),
)
print(response.json())
This will return the entire HTML content of the Indeed homepage. From here, you can adjust your scraping instructions to pull out specific data points.
Extracting Targeted Job Listings from Indeed
Now that you understand how the API works, let’s scrape job postings. Start by inspecting the job listing structure on Indeed:
Open Job Search Results in Chrome:
Right-click any job listing and select "Inspect" to view the HTML structure.
Find the CSS Selector for Job Listings:
Look for the class name that represents each job posting. For Indeed, it's .job_seen_beacon.
Here’s how you can create the payload to scrape job titles, company names, and more:
{
"source": "universal",
"url": "https://www.indeed.com/jobs?q=work+from+home&l=San+Francisco%2C+CA",
"render": "html",
"parse": True,
"parsing_instructions": {
"job_listings": {
"_fns": [
{
"_fn": "css",
"_args": [".job_seen_beacon"]
}
],
"_items": {
"job_title": {
"_fns": [
{
"_fn": "xpath_one",
"_args": [".//h2[contains(@class,'jobTitle')]/a/span/text()"]
}
]
},
"company_name": {
"_fns": [
{
"_fn": "xpath_one",
"_args": [".//span[@data-testid='company-name']/text()"]
}
]
}
}
}
}
}
Saving Data
Once you’ve scraped the data, you’ll want to save it. The Scraper API returns results in JSON, but with Pandas, you can quickly convert that into CSV:
import pandas as pd
# Extract job data from the response
df = pd.DataFrame(response.json()["results"][0]["content"]["job_listings"])
# Save the data as a CSV file
df.to_csv("job_search_results.csv", index=False)
Scraping Without an API (Using Proxies)
If you prefer a more hands-on approach without using the API, you can scrape job data using Python’s requests and residential proxies.
Install Required Libraries:
pip install requests beautifulsoup4
Set Up Proxies and Send Requests:
import requests
from bs4 import BeautifulSoup
# Use proxy credentials
USERNAME = 'PROXY_USERNAME'
PASSWORD = 'PROXY_PASSWORD'
# Define your proxies and Indeed URL
proxies = {
'http': f'http://{USERNAME}:{PASSWORD}@proxy.example.com:7777',
'https': f'https://{USERNAME}:{PASSWORD}@proxy.example.com:7777'
}
query = 'data scientist'
location = 'New York'
indeed_url = f'https://www.indeed.com/jobs?q={query.replace(" ", "+")}&l={location.replace(" ", "+")}'
# Send request and parse the response
response = requests.get(indeed_url, proxies=proxies)
soup = BeautifulSoup(response.text, 'html.parser')
# Extract job data
results = []
for job_card in soup.find_all('div', class_='job_seen_beacon'):
title = job_card.find('h2', class_='jobTitle').text.strip()
company = job_card.find('span', class_='companyName').text.strip()
link = job_card.find('a')['href']
results.append({'title': title, 'company': company, 'link': f'https://www.indeed.com{link}'})
# Print results
for result in results[:5]: # Limit to first 5 results
print(result)
Save to CSV:
import csv
# Save job data to a CSV
with open('indeed_results.csv', 'w', newline='', encoding='utf-8') as file:
writer = csv.writer(file)
writer.writerow(['Title', 'Company', 'Link'])
for result in results:
writer.writerow([result['title'], result['company'], result['link']])
Comparing Scraping Methods
When scraping data, there are different approaches, each with its own features, advantages, and limitations.
No proxies involve basic HTTP requests from a single IP address. This method is easy to implement and has no extra costs, but it faces challenges like blocked requests and lack of geo-targeting. It's best suited for small projects or testing.
On the other hand, with proxies involves rotating IPs and geo-location targeting. This approach offers a high success rate, scalability, and anti-ban features, making it ideal for large-scale scraping or competitor monitoring. However, it requires proxy management and comes with extra costs.
Finally, Scraper APIs are pre-built solutions that handle CAPTCHAs and parse data. They are ready-to-use, maintenance-free, and save time, though they can be more expensive and offer limited customization. These are best for scraping complex sites or JavaScript-heavy websites.
Conclusion
Web scraping Indeed job postings is powerful. With the right tools, you can automate the collection of job data, uncover valuable trends, and make smarter decisions. Whether you're using API or building your own scraper with proxies, the possibilities are endless.