Job postings data: how to read hiring trends from the web

· 2 min read · CrawlMine guides

Why job postings are a strong signal

Companies post jobs before they grow. A team that opens ten roles for a new product line is telling you about its plans months before a launch. Across a market, postings show which roles and skills are in demand, in which cities, at what pay where published. Recruiters, HR teams, educators, investors and policy researchers all use this signal.

Where the data comes from

  • Job boards, which give breadth across many employers
  • Company career pages, which give the original posting, often earlier and with more detail
  • Applicant tracking system pages, the public job pages many companies use
  • Public sector job portals

Collecting from boards and career pages together gives the best coverage, as long as you deduplicate.

Cleaning comes first

Deduplicate

The same opening is posted on several boards, reposted when it expires and sometimes listed in several cities. Without deduplication, counts are inflated. Match on company, title, location and description similarity, and keep the earliest date seen.

Normalise titles

“SDE II”, “Software Engineer, Backend” and “Developer (Java)” may describe similar roles. For trend analysis, keep the original title and add a normalised title and seniority. Roll-ups by role family become possible.

Extract skills

Skills sit in free-text descriptions. Extract them into a clean list using a skills vocabulary, so you can count how often each appears.

Standardise locations

Map city names and remote or hybrid status to a standard list.

Our job postings data service delivers postings cleaned in these ways.

What you can measure

  • Demand by role and city: open postings per normalised title per location
  • Skills demand: share of postings mentioning each skill, over time
  • Salary ranges: where published, by role and city
  • Company hiring momentum: postings per company per month
  • Remote share: proportion of roles offered as remote or hybrid

Common mistakes

  • Counting postings as hires. Postings are intent, not outcomes.
  • Ignoring duplicates. The most common source of inflated numbers.
  • Comparing raw titles. Without normalisation, trends are noise.
  • Treating published salaries as complete. Many postings do not show pay; the published share may not represent the whole market.
  • Short windows. Hiring is seasonal. Compare like periods.

Personal data

Job postings sometimes include a recruiter’s name or contact. We collect posting data and leave personal details out by default. Candidate data is never part of it.

Who uses it, and how

  • Recruitment agencies find companies hiring in their specialism. See recruitment and HR.
  • HR teams benchmark pay and see competition for talent.
  • Educators align courses with demanded skills. See education.
  • Investors read hiring momentum as a business signal. See alternative data.

Getting started

Define the roles, cities and sources that matter and start with a monthly or weekly feed. After three months you have a trend. For a sample, send us your roles and sites or email [email protected]. For model training or analytics, see web data for AI.

Talk to us directly

[email protected]

Send the websites and fields you need. A person on our team reads every message.

Email us

Send us the sites tonight. Data by morning.

Tell us which websites and fields you need. We confirm what is practical, how it should be delivered and what it costs.

  • Goes straight to [email protected]
  • Reply within one working day
  • A sample or pilot before any recurring plan
  • NDA available before you share details

[email protected]