Data scraping & pipelines tech stack 2026: what 36 real products use
Scraping, ETL, data integration and enrichment.
Among the 36 products we track in data scraping & pipelines, the usual stack is PostgreSQL (database, 35%), React (frontend framework, 54%), FastAPI (backend framework, 53%) and Vercel (hosting, 47%): shares of the products that show each decision.
How they're built
The typical stack
The leading pick where at least 8 of them show the decision and the leader has at least a quarter of it.
| Decision | Most common pick | Share | At default usage |
|---|---|---|---|
| Database | 9 of 26 · 35% | — | |
| Frontend Framework | 13 of 24 · 54% | — | |
| Backend Framework | 9 of 17 · 53% | — | |
| Hosting | 8 of 17 · 47% | $40/mo · Pro + Flat Rate CDN 10M | |
| LLM API | 8 of 14 · 57% | $10/mo · GPT-6 Luna | |
| AI SDK & Agent Framework | 4 of 13 · 31% | — | |
| Database Access & ORMs | 6 of 9 · 67% | — | |
| Transactional Email | 3 of 9 · 33% | $20/mo · Pro 50k | |
| Document Parsing | 4 of 8 · 50% | — |
Monthly bills add up to about $70 at the calculators' default usage, list prices. Set your own usage →
Where they differ from other products
Tools data scraping & pipelines pick at least twice as often as products overall. Each bar is the share of products that show the decision and chose the tool.
What they chose, decision by decision
Among the data scraping & pipelines that show each choice, from makers' products and open-source code alike. Under each tool, some of the products that chose it.
Small samples, fewer than 8 products: Background Jobs & Cron (Celery 3, RabbitMQ 2) · Vector Database (LanceDB 4, Qdrant 3) · Authentication (Clerk 2, Auth.js 2) · Image & Video Hosting (sharp 3, Cloudinary 1) · MCP Server for Your Product (MCP SDK 3, FastMCP 2) · Open Model Hosting & GPUs (Hugging Face 3, vLLM 3) · Payments (Stripe 4, Paddle 1) · AI UI Design (shadcn/ui 4) · Embeddings (Sentence Transformers 4) · Product Analytics (PostHog 4, Mixpanel 1) · Search APIs for AI (Tavily 3, Exa 1) · Web Scraping for AI (Scrapy 2, Crawl4AI 1)
Data scraping & pipelines we track
19 makers' products and 17 open-source projects, makers' products first, each with the tools it uses (the most common here first).
Questions
What tech stack is behind data scraping & pipelines?
Among the 36 products we track in data scraping & pipelines, the usual stack is PostgreSQL (database, 35%), React (frontend framework, 54%), FastAPI (backend framework, 53%) and Vercel (hosting, 47%): shares of the products that show each decision. After those, OpenAI API leads LLM API, LangChain leads AI SDK & agent framework and SQLAlchemy leads database access & ORMs.
Which tools are more common in data scraping & pipelines than elsewhere?
DuckDB for database (6.6× as often as among products overall), LiteLLM for LLM API (4.1× as often as among products overall) and MySQL for database (3.5× as often as among products overall).
How much does a typical data scraping & pipelines stack cost?
About $70 a month in bills at the pricing calculators' default usage and list prices. The biggest lines: Vercel $40, Resend $20 and OpenAI API $10.
Which products in data scraping & pipelines are open source?
17 of the 36 are open-source projects, known from their public code, among them HeadlessX, sparrow, NeMo-Retriever, data-prep-kit, DataFlow and lotus.