Updated
What makers build

Data scraping & pipelines tech stack 2026: what 36 real products use

Scraping, ETL, data integration and enrichment.

Among the 36 products we track in data scraping & pipelines, the usual stack is PostgreSQL (database, 35%), React (frontend framework, 54%), FastAPI (backend framework, 53%) and Vercel (hosting, 47%): shares of the products that show each decision.

How they're built

The typical stack

The leading pick where at least 8 of them show the decision and the leader has at least a quarter of it.

DecisionMost common pickShareAt default usage
DatabasePostgreSQL9 of 26 · 35%—
Frontend FrameworkReact13 of 24 · 54%—
Backend FrameworkFastAPI9 of 17 · 53%—
HostingVercel8 of 17 · 47%$40/mo · Pro + Flat Rate CDN 10M
LLM APIOpenAI API8 of 14 · 57%$10/mo · GPT-6 Luna
AI SDK & Agent FrameworkLangChain4 of 13 · 31%—
Database Access & ORMsSQLAlchemy6 of 9 · 67%—
Transactional EmailResend3 of 9 · 33%$20/mo · Pro 50k
Document ParsingPyMuPDF4 of 8 · 50%—

Monthly bills add up to about $70 at the calculators' default usage, list prices. Set your own usage →

Where they differ from other products

Tools data scraping & pipelines pick at least twice as often as products overall. Each bar is the share of products that show the decision and chose the tool.

What they chose, decision by decision

Among the data scraping & pipelines that show each choice, from makers' products and open-source code alike. Under each tool, some of the products that chose it.

Database26 products
PostgreSQL9 · 35%jitsu, cocoindex, Daft
MySQL9 · 35%cocoindex, Daft, DataFlow
MongoDB6 · 23%Scrapingdog, jitsu, web-page-monitor
DuckDB5 · 19%Daft, NeMo-Retriever, data-prep-kit
React13 · 54%jitsu, deepcrawl, Daft
Next.js10 · 42%Airtop, Scrapingdog, Parseflow.io
Astro5 · 21%Scrapingdog, Saturn, cocoindex
React Router4 · 17%jitsu, unstract, eyes
FastAPI9 · 53%docetl, HeadlessX, sparrow
Express5 · 29%Scrapingdog, jitsu, HeadlessX
Django3 · 18%unstract, zato, weibospider
Hono1 · 6%deepcrawl
Hosting17 products
Vercel8 · 47%Airtop, DataFuel.dev, Parseflow.io
Cloudflare4 · 24%PulpMiner, Saturn, deepcrawl
GitHub Pages2 · 12%NeMo-Retriever, PyAirbyte
LLM API14 products
OpenAI API8 · 57%Airtop, DataFuel.dev, cocoindex
LiteLLM6 · 43%cocoindex, unstract, docetl
Gemini API3 · 21%Saturn, Daft, DataFlow
LangChain4 · 31%fileAI, Tilores entity resolution, xberg
LlamaIndex4 · 31%unstract, xberg, data-prep-kit
AI SDK2 · 15%deepcrawl, docetl
Instructor2 · 15%cocoindex, sparrow
SQLAlchemy6 · 67%Daft, lotus, eyes
Prisma2 · 22%jitsu, HeadlessX
PyMuPDF4 · 50%docetl, xberg, DataFlow
Docling2 · 25%docetl, data-prep-kit
MarkItDown2 · 25%docetl, NeMo-Retriever
pypdf2 · 25%sparrow, DataFlow

Small samples, fewer than 8 products: Background Jobs & Cron (Celery 3, RabbitMQ 2) · Vector Database (LanceDB 4, Qdrant 3) · Authentication (Clerk 2, Auth.js 2) · Image & Video Hosting (sharp 3, Cloudinary 1) · MCP Server for Your Product (MCP SDK 3, FastMCP 2) · Open Model Hosting & GPUs (Hugging Face 3, vLLM 3) · Payments (Stripe 4, Paddle 1) · AI UI Design (shadcn/ui 4) · Embeddings (Sentence Transformers 4) · Product Analytics (PostHog 4, Mixpanel 1) · Search APIs for AI (Tavily 3, Exa 1) · Web Scraping for AI (Scrapy 2, Crawl4AI 1)

Data scraping & pipelines we track

19 makers' products and 17 open-source projects, makers' products first, each with the tools it uses (the most common here first).

xbergPolyglot document intelligence with a Rust core: extract text, metadata, images, tables, and structured data from 106 formats across 140 file extensions, plus code intelligence for 371 languages. FiftAstroLangChainLlamaIndexPyMuPDFsharp
datacapDataCap is integrated software for data transformation, integration, and visualization. Support a variety of data sources, file types, big data related database, relational database, NoSQL database, eMySQLClickHouseVue.js

Questions

What tech stack is behind data scraping & pipelines?

Among the 36 products we track in data scraping & pipelines, the usual stack is PostgreSQL (database, 35%), React (frontend framework, 54%), FastAPI (backend framework, 53%) and Vercel (hosting, 47%): shares of the products that show each decision. After those, OpenAI API leads LLM API, LangChain leads AI SDK & agent framework and SQLAlchemy leads database access & ORMs.

Which tools are more common in data scraping & pipelines than elsewhere?

DuckDB for database (6.6× as often as among products overall), LiteLLM for LLM API (4.1× as often as among products overall) and MySQL for database (3.5× as often as among products overall).

How much does a typical data scraping & pipelines stack cost?

About $70 a month in bills at the pricing calculators' default usage and list prices. The biggest lines: Vercel $40, Resend $20 and OpenAI API $10.

Which products in data scraping & pipelines are open source?

17 of the 36 are open-source projects, known from their public code, among them HeadlessX, sparrow, NeMo-Retriever, data-prep-kit, DataFlow and lotus.