This video is available to ERE Pro subscribers

Subscribe to watch

Already a member? Log in

Choose-your-own-adventure- Build your own no-code, automated, search scraper

October 22, 2024 · SourceCon Fall 2024 ·

Speakers

About this video

A practical framework shows how to stack free and cheap tools into an automated pipeline that catches job postings and candidate profiles the moment they appear online, without writing a line of code.

The approach was born out of a real problem: during heavy layoffs, volunteers were manually searching job boards and sharing links, but postings filled so fast that the links were often dead by the time anyone clicked. The fix was to automate the alerting instead of the searching.

The system is built like a pyramid:

  • At the base, dozens of Google Alerts are set up using site-pattern searches (like jobs.ashby.com) combined with role keywords and excluded terms to filter out false positives, such as procurement listings showing up under "sourcer" searches.
  • RSS and Atom feeds from job boards and applicant tracking systems add a second layer of coverage, including niche sources like compliance-feed aggregators that collect postings companies are required to share with government and disability employment channels.
  • An RSS reader tool ties everything together, deduplicating results, applying regular expressions instead of clunky Boolean strings, and using webhooks to push clean results into a spreadsheet within minutes of a job going live.
  • Google Sheets serves as the final output layer, tracking thousands of postings by source and exact posting time, with tabs organized by ATS platform and by individual job board.

Regular expressions, generated with help from an AI tool, replace error-prone Boolean strings and cut down on the extra spaces and missing quotation marks that quietly wreck search results. Timestamp data gets converted from raw epoch format into readable, sortable dates so postings can be analyzed by day, hour, or source.

The same structure applies just as well to sourcing candidates as it does to sourcing jobs. Swapping job-board patterns for resume-hosting sites and adjusting keywords (title, location, credential) turns the identical alert-and-scrape pipeline into a way to catch newly updated resumes and profiles, useful for licensed professionals such as engineers who rarely refresh their online presence but do so at meaningful moments.

The tools involved cost only a few dollars a month, run without any coding knowledge, and scale from a single city search to dozens of markets by duplicating tabs and swapping location terms.