Web Scraping for AI
Turn web pages into clean Markdown or structured data that a model can read, for RAG, research agents or data features.
Trends
Firecrawl gained the most ground in package downloads: +100% against the median npm package over a year.
Public counts each month: pull requests on GitHub, package downloads, stars, Product Hunt followers. They show direction, not market share.
The real choice
Hosted API or library you run. Hosted APIs handle JavaScript rendering, proxies and blocking for per-page credits. Running an open-source crawler yourself costs only servers, but blocked requests and broken pages become your problem.
Pick by situation
Tap the ones that are you — the tools that fit light up below.- Your agent reads single pages it found through search
Jina AI
Firecrawl - You need whole sites crawled into Markdown or schema-shaped JSON without running browsers
Firecrawl
Context.dev - Someone has already built a scraper for the exact site you need
Apify - You have steady volume and would rather run the crawler on your own servers
Crawl4AI
CrawleeScrapy
Compare the tools
Grouped by the side of the choice they answer, not ranked. Open a row for when to use it, the trade-off and what makers say.
Firecrawlapi · open sourceCrawling whole sites into Markdown or JSON through one API that handles rendering and blocking.


26+45 open source1.2M/wk+100% vs npm
Use it whenYou need many pages from a site, or structured fields pulled out with a schema.
Trade-offCredits add up on large crawls; self-hosting the open-source version lacks some cloud features.
In LovableUsed by adomate, Agentplace.io, Astrio, Hivinq, Locus Founder and 21 more · in 45 open-source projects.
We point Firecrawl at a brand's site the moment someone adds the URL, and it feeds straight into their Brand Brain. No scraper to build, no maintenance, just clean data on the way in.
Context.devapiScraping and crawling pages into Markdown plus company brand data — logos, colors, fonts — from one credit-based API.


18
Use it whenYour product needs both page content and company information, such as onboarding that brands itself from a URL.
Trade-offA younger service than the larger scraping APIs; credits cover every call, including browser actions and OCR.
Used by Click, Construct Computer, DocsAlot, Hoplite, Migma AI and 13 more.
We're using Context.dev to enrich the profiles in our graph. When NepoNet finds an intro path, knowing someone's company, portfolio, what they do, makes the suggested intro much more useful.
Jina AIapiTurning a single URL into Markdown by adding a prefix to it, with no setup or key needed to try.No evidence yet
Use it whenYour agent reads individual pages it found through search.
Trade-offBuilt for one page at a time rather than crawling whole sites; keyless use is rate-limited.
No maker's product we track shows it for this yet.
ApifyplatformReady-made scrapers for specific sites — maps listings, social profiles, e-commerce — that you run without writing one.


13+13 open source370.7k/wk+26% vs npm
Use it whenSomeone has already built a scraper for the exact site you need.
Trade-offCommunity scrapers vary in quality and break when the target site changes.
In LovableUsed by Click, MindPal, Mito Health, Origami.chat, Pancake and 8 more · in 13 open-source projects.
Apify's infrastructure and prebuilt scrapers power a lot of my connectors, and help me focus on building an amazing experience for the rest of my product.
Scrapyframework · open sourceA mature Python framework for fast, large crawls of mostly static sites, with a big ecosystem of plugins.28 open source995.8k/wk
Use it whenYou crawl many pages from sites that don't need JavaScript to render.
Trade-offJavaScript-heavy sites need a browser plugin, and it has a learning curve of its own.
No maker's product we track shows it for this yet · in 28 open-source projects.
Crawl4AIlibrary · open sourceA free Python crawler that outputs LLM-ready Markdown on your own machines.20 open source294.3k/wk
Use it whenYou have steady volume and would rather pay for servers than per page.
Trade-offYou handle proxies, blocking and browser upkeep yourself.
No maker's product we track shows it for this yet · in 20 open-source projects.
Crawleelibrary · open sourceAn open-source crawling library for Node.js and Python with queues, retries, proxy rotation and headless browsers built in.3 open source152.4k/wk+6% vs npm
Use it whenYou write your own crawlers and want the plumbing handled, with the option to run them on Apify later.
Trade-offYou still run the servers and browsers and handle blocking; LLM-ready Markdown output is up to you.
No maker's product we track shows it for this yet · in 3 open-source projects.
Before you choose
Start with the simplest thing that returns clean Markdown for one URL, and only add crawling when you know which pages you need. Cache everything you fetch so reruns don't cost credits or hit sites again. Respect robots.txt and site terms; logins and paywalls are a legal question before they're a technical one.
- Fetching the same pages on every request instead of caching them, paying credits twice and hammering the site.
- Feeding raw HTML to the model, spending most of the tokens on navigation, scripts and markup instead of content.
- Building a feature on scraping one site without checking its terms, then losing it to a block or a letter from their lawyers.
Other options
Real choices most makers here won't need to weigh.
Decided alongside
What the 54 makers' products here chose for their other decisions.