Updated
Agent building blocks

Web Scraping for AI

Turn web pages into clean Markdown or structured data that a model can read, for RAG, research agents or data features.

For your framework or language:Next.js
Ask your AI about this, with this page as the source:ChatGPT ↗Claude ↗Perplexity ↗

Trends

Firecrawl gained the most ground in package downloads: +100% against the median npm package over a year.

Public counts each month: pull requests on GitHub, package downloads, stars, Product Hunt followers. They show direction, not market share.

ToolDownloadsStars
Firecrawl1.2M/wk+100% vs npm★ 192.1k
Scrapy995.8k/wk★ 64.8k
Apify370.7k/wk+26% vs npm—
Crawl4AI294.3k/wk★ 85.3k
Crawlee152.4k/wk+6% vs npm★ 26.2k
Bright Data8.9k/wk−20% vs npm★ 2.7k
Spider3.7k/wk−64% vs npm★ 2.8k
ScrapeGraphAI2.1k/wk★ 31.9k
Diffbot44/wk—

The real choice

Hosted API or library you run. Hosted APIs handle JavaScript rendering, proxies and blocking for per-page credits. Running an open-source crawler yourself costs only servers, but blocked requests and broken pages become your problem.

What makers chose54 live products, each with a source. Not market share.

Pick by situation

Tap the ones that are you — the tools that fit light up below.
  • Your agent reads single pages it found through searchJina AIFirecrawl
  • You need whole sites crawled into Markdown or schema-shaped JSON without running browsersFirecrawlContext.dev
  • Someone has already built a scraper for the exact site you needApify
  • You have steady volume and would rather run the crawler on your own serversCrawl4AICrawleeScrapy

Compare the tools

Grouped by the side of the choice they answer, not ranked. Open a row for when to use it, the trade-off and what makers say.

ToolBest for
Hosted API3
Firecrawlapi · open sourceCrawling whole sites into Markdown or JSON through one API that handles rendering and blocking.26+45 open source1.2M/wk+100% vs npm

Use it whenYou need many pages from a site, or structured fields pulled out with a schema.

Trade-offCredits add up on large crawls; self-hosting the open-source version lacks some cloud features.

In Lovable

Used by adomate, Agentplace.io, Astrio, Hivinq, Locus Founder and 21 more · in 45 open-source projects.

We point Firecrawl at a brand's site the moment someone adds the URL, and it feeds straight into their Brand Brain. No scraper to build, no maintenance, just clean data on the way in.
adomate, the makerSep 2026 ↗
Context.devapiScraping and crawling pages into Markdown plus company brand data — logos, colors, fonts — from one credit-based API.18

Use it whenYour product needs both page content and company information, such as onboarding that brands itself from a URL.

Trade-offA younger service than the larger scraping APIs; credits cover every call, including browser actions and OCR.

Used by Click, Construct Computer, DocsAlot, Hoplite, Migma AI and 13 more.

We're using Context.dev to enrich the profiles in our graph. When NepoNet finds an intro path, knowing someone's company, portfolio, what they do, makes the suggested intro much more useful.
Nepotism Network, the makerSep 2026 ↗
Jina AIapiTurning a single URL into Markdown by adding a prefix to it, with no setup or key needed to try.No evidence yet

Use it whenYour agent reads individual pages it found through search.

Trade-offBuilt for one page at a time rather than crawling whole sites; keyless use is rate-limited.

No maker's product we track shows it for this yet.

Scraper marketplace1
ApifyplatformReady-made scrapers for specific sites — maps listings, social profiles, e-commerce — that you run without writing one.13+13 open source370.7k/wk+26% vs npm

Use it whenSomeone has already built a scraper for the exact site you need.

Trade-offCommunity scrapers vary in quality and break when the target site changes.

In Lovable

Used by Click, MindPal, Mito Health, Origami.chat, Pancake and 8 more · in 13 open-source projects.

Apify's infrastructure and prebuilt scrapers power a lot of my connectors, and help me focus on building an amazing experience for the rest of my product.
Click, the makerSep 2026 ↗
Library you run3
Scrapyframework · open sourceA mature Python framework for fast, large crawls of mostly static sites, with a big ecosystem of plugins.28 open source995.8k/wk

Use it whenYou crawl many pages from sites that don't need JavaScript to render.

Trade-offJavaScript-heavy sites need a browser plugin, and it has a learning curve of its own.

No maker's product we track shows it for this yet · in 28 open-source projects.

Crawl4AIlibrary · open sourceA free Python crawler that outputs LLM-ready Markdown on your own machines.20 open source294.3k/wk

Use it whenYou have steady volume and would rather pay for servers than per page.

Trade-offYou handle proxies, blocking and browser upkeep yourself.

No maker's product we track shows it for this yet · in 20 open-source projects.

Crawleelibrary · open sourceAn open-source crawling library for Node.js and Python with queues, retries, proxy rotation and headless browsers built in.3 open source152.4k/wk+6% vs npm

Use it whenYou write your own crawlers and want the plumbing handled, with the option to run them on Apify later.

Trade-offYou still run the servers and browsers and handle blocking; LLM-ready Markdown output is up to you.

No maker's product we track shows it for this yet · in 3 open-source projects.

Before you choose

How to approach it

Start with the simplest thing that returns clean Markdown for one URL, and only add crawling when you know which pages you need. Cache everything you fetch so reruns don't cost credits or hit sites again. Respect robots.txt and site terms; logins and paywalls are a legal question before they're a technical one.

Common mistakes
  • Fetching the same pages on every request instead of caching them, paying credits twice and hammering the site.
  • Feeding raw HTML to the model, spending most of the tokens on navigation, scripts and markup instead of content.
  • Building a feature on scraping one site without checking its terms, then losing it to a block or a letter from their lawyers.

Other options

Real choices most makers here won't need to weigh.

Decided alongside

What the 54 makers' products here chose for their other decisions.