Web Scraping
SaaS, open source and skills for web scraping work — with the limits of each written down.Before you pick
What decides it
Your hardest target sets the bill: a page needing a browser and a residential exit costs many times a plain fetch.
What to avoid
Blocks rarely arrive as errors: a challenge page returns 200, the model extracts confident fields from it, and the pipeline records success over nothing.
What is changing
Sites are shifting from blocking to identifying: crawlers present signed keys, and the answer is increasingly a price per fetch rather than a refusal.
SaaS
Hosted products you sign up for.Leading
Established picks, at the top of today's aggregate.
- 1FirecrawlTurning any URL into clean, LLM-ready Markdown or schema-defined JSON in a single call — the default for RAG and AI pipelines.
- 2Bright DataMaximum-scale, hard-target scraping via the Unlocker API, Agent Browser and the broadest proxy infrastructure available.
- 3ApifyA mature developer platform with 6,000+ prebuilt scrapers (Actors), scheduling and proxy management for complex, custom pipelines.
- 4Browse AINo-code scraping and scheduled change-monitoring of any site for non-technical users.
- 5OctoparseA visual, point-and-click desktop tool that handles complex extraction with zero code.
Emerging
Newer challengers, surfaced by two or more sources.
- 1Scraping FishScraping API that returns rendered HTML through real browsers and rotating mobile proxies, billed per successful request with no subscription or account.
- 2CrawlbaseNow Crawlbase: a crawling API with anti-bot handling, rotating residential proxies, site-specific scrapers and an MCP endpoint for agents.
Open source
Repositories you run yourself.Leading
- 1Crawl4AIThe de-facto king of open-source AI web scraping (~74.8k★) — LLM-friendly crawler turning pages into clean Markdown/JSON for RAG and agent pipelines.
- 2ScrapyThe classic, most battle-tested production Python crawling framework (~63.4k★) — unmatched for large, structured crawls at scale.
- 3PlaywrightMicrosoft's cross-browser automation (~93k★) — the modern default for scraping JS-heavy, dynamically-rendered sites.
- 4PuppeteerGoogle's headless Chrome automation (~95k★) — ubiquitous and extremely mature for browser-driven scraping.
- 5Beautiful SoupThe universally-adopted Python HTML/XML parser (beautifulsoup4, one of the most-downloaded packages ever) — the default for simple, static-page extraction.
Emerging
- 1oxylabs/ai-scraper-pyPython client for Oxylabs' hosted AI Studio extraction API: describe the fields you want in plain English, get JSON back.
- 2ScrapingBee/n8n-no-code-web-scraperA tutorial and example n8n workflow JSON that wires ScrapingBee's API and an LLM into a scheduled scraping pipeline; not a library.
- 3ScrapeGraphAI/just-scrapeNode CLI over the ScrapeGraph hosted API, with subcommands for markdown scraping, schema-driven JSON extraction, crawling and cron page-change monitoring.
- 4n24q02m/wet-mcpMCP server giving agents web search and page extraction with no API keys, escalating from plain HTTP to browser rendering when blocked.
- 5supadata-ai/skillsWraps the Supadata API so an agent can pull video transcripts and metadata, or scrape and crawl pages into Markdown.
Skills
Skills you drop into a coding agent.Leading
- 1Scrapling Skill (Cedriccmh)The most-starred standalone scraping Claude Code skill (~372★) — a SKILL.md wrapping Scrapling with automatic fetcher selection and Cloudflare bypass.
- 2Web Scraping skills (Mindrally)Two genuine SKILL.md scraping skills (web-scraping + scrapy-web-scraping) inside a 240+ Claude Code skill collection (~197★).
Emerging
- 1gosom/google-maps-scraperGo and Playwright scraper that pulls business listings from Google Maps — name, address, phone, hours, reviews, coordinates — via CLI, web UI or REST.
- 2yfe404/web-scraperClaude Code skill that probes a site with curl first, preferring hidden APIs and sitemaps over DOM scraping, then ships an Apify actor.
- 3Lunatic16/paywall-bypass-skillFingerprints paywall providers like Piano and Poool from page source, then suggests extraction routes such as AMP pages, JSON-LD or archive.is.
- 4thirdwatch-dev/scraping-skillsMarkdown skills that teach coding agents how to approach anti-bot targets, then route common jobs to the author's pay-per-result Apify scrapers.
- 5Decodo/agent-skillsTwo vendor skills that tell coding agents when and how to call Decodo's scraping API, CLI or MCP server, including price-monitoring workflows.