Job scraping: how to collect job postings and which sites allow it

Job postings live in two places. Job boards collect them, and companies publish them at the source, usually through the hiring system behind their careers page. Most big boards forbid scraping. Several hiring systems publish every open posting through a public API on purpose.

On 12 October 2026 we read the robots.txt files of 15 job sites and the terms of Indeed and LinkedIn. Then we collected postings through two hiring systems' public APIs. Every request ran from our own server, and we fetched no page that a robots.txt file forbids.

The short answer

• Collect at the source first. Greenhouse's and Lever's public APIs returned 18 and 10 postings in one request each, with no key.
• Use official job APIs where they exist. USAJOBS and Adzuna give keys to anyone who registers.
• Read a board's robots.txt and terms before anything else. LinkedIn and ZipRecruiter disallow every crawler they do not name. Indeed's terms bar automated access without written permission.
• Read robots.txt with a parser written to the standard. Python's built-in reader got 4 of 26 job addresses wrong in our test.
• A proxy changes your address, not a site's rules.

What 15 job sites' robots.txt files say

For each site we asked about two addresses in the site's own format, a job search and a single job. We asked for a crawler without its own rules and for Googlebot. We fetched only the robots.txt files.

Six of the 13 files we could read restrict job addresses.

LinkedIn's file is 120,190 bytes long, with 76 groups of rules. Googlebot may crawl LinkedIn's and ZipRecruiter's job pages, and other crawlers may not. Glassdoor, Monster, Wellfound, USAJOBS and the job search of Germany's Federal Employment Agency allowed both addresses. So did the job pages that Greenhouse and Lever host for their customers. SimplyHired answered our request for its robots.txt with 403, and StepStone did not answer within 60 seconds.

An allowed address is not permission. The robots.txt standard, RFC 9309, says it plainly. "These rules are not a form of access authorization". A site's terms can forbid what its robots.txt allows.

What the terms say

Indeed's terms tie the two together. They bar "any automated system (bots, scrapers, spiders, AI or Agentic AI)" from the site without Indeed's "express written permission". In brackets they add "we conditionally grant permission to crawl the Site solely as outlined in our robots.txt file". The next sentence is broader. "You may not crawl, scrape, extract data from, reproduce, duplicate, copy, sell, exploit, trade or resell any part of the Site".

LinkedIn's User Agreement bars software, scripts and robots "to scrape or copy the Services, including profiles and other data from the Services". Glassdoor's and ZipRecruiter's terms pages answered our server with 403, so we could not read them.

Our pages on Indeed, LinkedIn and Google Jobs go through each site's rules. They also cover the official data each one offers.

Collect at the source: public job APIs

Companies that hire through Greenhouse or Lever publish their open jobs through public APIs. Greenhouse's documentation says "Job Board data is publicly available, so authentication is not required for any GET endpoints". Lever's says "all job postings in the published state are publicly viewable". It adds "These jobs may be scraped by third parties".

import requests

HEADERS = {"User-Agent": "job-collector/1.0 (+https://example.com/contact)"}
board = "greenhouse"  # from the board's address: job-boards.greenhouse.io/greenhouse
r = requests.get(f"https://boards-api.greenhouse.io/v1/boards/{board}/jobs",
                 params={"content": "true"}, headers=HEADERS, timeout=30)
r.raise_for_status()
jobs = r.json()["jobs"]
print(len(jobs), "postings")
for job in jobs[:3]:
    print(job["title"], "|", job["location"]["name"], "|", job["updated_at"][:10])

18 postings
Commercial Counsel | Eastern Timezone | 2026-10-09
Enterprise Sales Development Representative | Anywhere in the United States | 2026-10-09
Growth Account Executive | Anywhere in Ireland | 2026-09-02

Greenhouse's own board came back in one response of 220,522 bytes. Each posting carried its full description, location, departments and dates. The board name comes from the board's address, so a company's board at job-boards.greenhouse.io/name is read with the name name.

Lever works the same way. The site name below is the demo site that Lever's own documentation uses.

site = "leverdemo"  # the demo site Lever's own documentation uses
r = requests.get(f"https://api.lever.co/v0/postings/{site}", params={"mode": "json"}, headers=HEADERS, timeout=30)
r.raise_for_status()
postings = r.json()
print(len(postings), "postings")
for p in postings[:3]:
    pay = p.get("salaryRange")
    pay_text = f"{pay['min']} to {pay['max']} {pay['currency']}" if pay else "no pay range"
    print(p["text"], "|", p["categories"].get("location"), "|", pay_text)

10 postings
Customer Success Manager | Atlanta, Georgia | 85000 to 175000 USD
Dentist | Houston, TX | 150000 to 185000 USD
Director of Diversity, Equity, and Inclusion | London, UK | no pay range

Lever's own site returned an empty list that day, so the demo site stands in. Neither API sent rate-limit headers. Lever's job pages ask for a Crawl-delay of 1 second, and we kept it for its API too.

Job postings inside the page: JobPosting data

Google shows job postings in its job search when a page carries schema.org JobPosting data. Its guide lists five required properties: datePosted, description, hiringOrganization, jobLocation and title. Where a page carries that block, a parser can read it without guessing at the layout.

import json
import time

from bs4 import BeautifulSoup

time.sleep(1)  # jobs.lever.co asks for a Crawl-delay of 1 second
page = requests.get(postings[0]["hostedUrl"], headers=HEADERS, timeout=30)
soup = BeautifulSoup(page.text, "lxml")
for tag in soup.find_all("script", type="application/ld+json"):
    data = json.loads(tag.string)
    if data.get("@type") == "JobPosting":
        print(data["title"], "|", data["datePosted"], "|", data["hiringOrganization"]["name"])
        print(sorted(data))

Customer Success Manager | 2026-08-31 | Lever Demo 2
['@context', '@type', 'datePosted', 'description', 'employmentType', 'hiringOrganization', 'jobLocation', 'title']

Lever's job page carried all five required properties. The page Greenhouse hosts for its own job carried no JobPosting block, even after its scripts ran in Chrome. So read the API where a company has one, and use JobPosting data where a page carries it.

Read robots.txt the way the standard does

Python's built-in urllib.robotparser and Protego disagreed on 4 of our 26 job addresses. Protego is the "Pure-Python robots.txt parser with support for modern conventions" that Scrapy uses.

import urllib.robotparser

from protego import Protego

robots = requests.get("https://www.indeed.com/robots.txt", headers=HEADERS, timeout=30).text
url = "https://www.indeed.com/viewjob?jk=1a2b3c4d5e6f7a8b"

builtin = urllib.robotparser.RobotFileParser()
builtin.parse(robots.splitlines())
print("urllib.robotparser:", builtin.can_fetch("job-collector", url))
print("protego (RFC 9309):", Protego.parse(robots).can_fetch(url, "job-collector"))

urllib.robotparser: True
protego (RFC 9309): False

Indeed's rules for all crawlers start with Allow: /, and Disallow: /viewjob comes later. The standard says "The most specific match found MUST be used", so the longer rule wins and the job page is off limits. Python's reader returns the first rule that matches, and its own source compares plain prefixes with filename.startswith(self.path).

The other three disagreements followed from the same two habits. Python allowed CareerBuilder's job pages and Welcome to the Jungle's search, whose rule uses * wildcards. It refused Dice's job pages, which the standard allows. If your crawler checks robots.txt, check it with a parser like Protego.

Official job APIs with a key

USAJOBS is "The Federal Government's official employment site". Its developer guide says "Accessing the various USAJOBS APIs will require an API Key", and it wants your email address as the User-Agent. Adzuna's API asks you to "Register to receive an app_key and app_id". We did not register for either, so we did not test them.

Where a proxy fits

Public APIs, and pages a robots.txt file allows, need no proxy. A proxy changes the address a site sees and nothing in its terms. Our page on whether web scraping is legal covers the court cases behind these rules.

What this page could not check

• Each robots.txt file was read once. SimplyHired refused our server and StepStone did not answer.
• The test addresses follow each site's own link format. A site may use other formats as well.
• We read the terms of Indeed and LinkedIn. Glassdoor and ZipRecruiter refused our server.
• Two hiring systems were tested. The documentation of four more (Ashby, SmartRecruiters, Recruitee and Personio) asked our server to prove it was human, so we left them out.
• This page is not legal advice.
• We will check this page again by 12 January 2027.

Sources

• Greenhouse, Job Board API: developers.greenhouse.io.
• Lever, Postings API README: github.com.
• Indeed, Terms of Service: indeed.com.
• LinkedIn, User Agreement: linkedin.com.
• The robots.txt files of the 15 sites in the table and the capture, read on our server.
• RFC 9309, Robots Exclusion Protocol: rfc-editor.org.
• CPython 3.12, Lib/urllib/robotparser.py: github.com.
• Protego 0.7.0 on PyPI: pypi.org.
• Google Search Central, job posting structured data: developers.google.com.
• schema.org, JobPosting: schema.org.
• USAJOBS developer guide, authentication: developer.usajobs.gov.
• Adzuna API overview: developer.adzuna.com.
• Our own tests on our server, 12 October 2026: boards_check.py at 08:33 UTC, robots_parsers.py at 08:34 UTC, ats_apis.py at 08:25 UTC and the code on this page at 08:32 UTC.