TL;DR — the honest scrape playbook
Scraping publicly accessible data with rate-limited requests, respect for robots.txt, and no ToS bypass is legal in most jurisdictions in 2026. Use Playwright for JS-heavy sites, requests + selectolax for static HTML. Add residential proxies (Bright Data / Smartproxy), exponential backoff on 429/503, and cache aggressively. If you scrape personal data, GDPR/CCPA apply regardless of where you're based — get a privacy notice and a deletion path.
Web scraping in 2026 sits at an uncomfortable intersection of legal but contested, technical but increasingly hard, and useful but easy to do badly. The hiQ v. LinkedIn rulings settled most of the US legal question for public data. But anti-bot layers (Cloudflare, PerimeterX, DataDome) have gotten dramatically better, and GDPR/CCPA enforcement has gotten dramatically stricter.
This is the practical guide we wish we'd had when we started — the legal lines you can't cross, the technical stack that actually works in 2026, and the ethical defaults that keep you out of court and out of IP blocklists.
In this post
1. The legal landscape (US, EU, UK)
US — Computer Fraud and Abuse Act (CFAA). After hiQ Labs v. LinkedIn (9th Circuit, 2022) and the Van Buren clarification, scraping publicly accessible data — pages anyone can view without authentication — is not "unauthorized access" under the CFAA. Scraping behind a login, bypassing IP blocks specifically targeted at you, or using fake accounts can still create exposure.
EU — GDPR. The data-protection question is independent of legality of access. If your scraped data includes any personal information about an EU person, you need a lawful basis (almost always "legitimate interest"), a publicly visible privacy notice naming the data source, and a way to honor erasure requests under Article 17.
UK — DPA 2018 + GDPR-equivalent. Largely tracks the EU model. The ICO has been active on scraping enforcement — 2024–2025 saw notable fines against scrapers of public LinkedIn data who didn't notify subjects.
Terms of Service. Even where scraping is technically and legally clear, sites can sue under contract law if their ToS expressly prohibits scraping AND you've assented to that ToS (by signing up). The 2nd Circuit's Meta v. Bright Data decision (2024) reaffirmed that browsewrap (no login, no acceptance click) is generally not enforceable; clickwrap (you actively agreed) is.
2. The ethical defaults
The legal floor is low. The ethical floor is higher and almost always the right call:
- Respect
robots.txtby default. It's a convention, not a contract — but it signals good faith and most production scrapers honor it. - Rate-limit to ≤ 1 request/second per domain unless you have explicit permission to go faster.
- Set a real
User-Agentidentifying your project + contact email. Don't pretend to be a browser when you're not. - Cache aggressively — never re-fetch an unchanged page. Use
If-Modified-Since/ETagheaders properly. - Stop the moment a site asks you to. If a site emails you a cease-and-desist for clearly public data, respond, evaluate, and likely comply. The legal cost of a fight rarely justifies the data.
3. Modern 2026 scraping stack
The stack depends on the site:
- Static HTML / server-rendered: Python
requests+selectolax(orlxml). 10–50× faster than a headless browser. Use for news sites, blogs, classifieds, most government data. - JS-rendered SPAs: Playwright (Python or Node). More reliable than Puppeteer in 2026, better Chromium/WebKit/Firefox parity. Pair with
playwright-stealthfor fingerprint masking. - Massive scale (10M+ pages): Scrapy in Python with Playwright integration, queued via Redis or AWS SQS. Workers shard by domain.
- Storage: Postgres for transactional/relational data; DuckDB or Parquet for analytics workloads; S3 + Athena for archive.
- Proxies: Bright Data (most expensive, best coverage), Smartproxy (mid-tier), Oxylabs (good geo distribution). Budget $200–$2,000/month for production volume.
- Monitoring: Sentry for errors, Grafana for queue depth + success rate, PagerDuty for alerting on anti-bot escalations.
Stuck on Cloudflare or DataDome?
Our scraping specialists ship production pipelines through every major anti-bot stack — with proper rate limits, monitoring, and compliance. Fixed-price quotes, 14-day delivery.
Hire a scraping expert →4. Anti-bot bypass without breaking ToS
The big anti-bot vendors in 2026 — Cloudflare Bot Management, PerimeterX (now HUMAN), DataDome, Akamai Bot Manager, Imperva — have moved past simple TLS fingerprinting into ML-based behavioral analysis. Bypass strategies that work:
- Residential proxies, NOT datacenter. Cloudflare scores datacenter IPs as high-risk by default.
- Real browser stacks (Playwright/Puppeteer with stealth plugins) over HTTP clients for any site behind serious anti-bot.
- Sensible mouse / scroll / keystroke patterns if you have to interact. Headless mode without any human-like noise is increasingly detected.
- Realistic session reuse. Don't kill cookies after every request — real users have sessions that last minutes to hours.
- Honor backoff signals. When a site returns 429 or 503, slow down dramatically. Most anti-bot systems are forgiving if you actually respect their signals.
If the site is protected AND its ToS bans scraping AND you signed up for an account — bypassing the protection is the legally risky combination. Public + no account + reasonable rate is the safe zone.
5. Handling scraped personal data (GDPR/CCPA)
If any scraped field looks like a person (name, email, photo, IP, profile URL), treat the data as personal under GDPR/CCPA:
- Document your lawful basis. For most scrapers it's "legitimate interest" — write down the balancing test (your interest vs the subjects' rights).
- Publish a privacy notice on your public website naming the data source and your processing purpose. Reachable in ≤ 2 clicks.
- Implement deletion. When a person emails you asking for their data to be removed (Article 17 / CCPA right to delete), respond within 30 days and actually remove it from your storage + caches + backups.
- Don't sell raw personal data. The moment you monetize personal data scraped without consent, you cross into a different enforcement category.
6. Anti-patterns that get you blocked or sued
- Hammering a site at 50 req/sec. This is the #1 cause of IP blocks AND civil lawsuits. Slow down.
- Faking residency to scrape geo-restricted content. Adds a fraud-by-deception layer to whatever else you're doing.
- Scraping behind a paid login then republishing. Both copyright AND contract violation.
- Scraping competitor SaaS dashboards. Almost always behind ToS-protected logins. Don't.
- Ignoring 429s and continuing. Signals bad faith. Courts notice. Anti-bot systems double-down.
7. DIY vs hire a specialist
DIY when:
- One-off project, under ~50K pages, public unprotected site
- You can read Python or JS and debug a failed selector
- Data doesn't include personal info
Hire a specialist when:
- Ongoing schedule with SLA + monitoring
- Anti-bot stack (Cloudflare, DataDome, PerimeterX)
- Personal data + compliance obligations
- Data flowing into a customer-facing product
- Volume past ~100K pages/day
FAQ
Is web scraping legal in 2026?
Scraping publicly accessible data is generally legal in the US after hiQ Labs v. LinkedIn (2022) and follow-up rulings. Scraping behind a login, bypassing technical barriers, or violating express ToS clauses still creates CFAA exposure. EU sites add GDPR considerations for any personal data. The honest answer: 'public + low rate + no ToS bypass' is safe; everything else needs a lawyer.
What's the best web scraping stack in 2026?
For most modern JS-heavy sites: Playwright in Python or Node, paired with a residential or datacenter proxy pool (Bright Data, Smartproxy, Oxylabs), a queue (Redis or AWS SQS), and a normalised storage layer (Postgres or DuckDB). For static HTML, requests + selectolax in Python is 10× faster than headless browsers.
Do I have to respect robots.txt?
Legally, robots.txt isn't binding in most jurisdictions (it's a convention, not a contract). Ethically and practically, yes — ignoring it gets your IPs blocked, kills your reputation with the site's ops team, and signals bad faith to courts if anything escalates. Production scrapers respect robots.txt by default; bypass requires explicit business justification.
How do I avoid getting blocked?
Five things: (1) rate-limit aggressively — 1 req/sec per domain is generally safe; (2) rotate residential proxies on a sensible cadence; (3) mimic real browser fingerprints (Playwright + playwright-stealth); (4) cache aggressively so you don't refetch unchanged pages; (5) handle 429/503 with exponential backoff. Most blocks come from being too fast, not from being detected as a bot.
What about GDPR / CCPA for scraped data?
If the scraped data includes any personal information about EU or California residents (names, emails, profile photos, IPs), GDPR/CCPA apply regardless of where you operate. You need a lawful basis (usually 'legitimate interest'), a public privacy notice, and a way to honor deletion requests. For anonymous aggregate data (prices, stock levels, public metrics), neither applies.
Should I hire a scraping specialist or DIY?
DIY for one-off projects under 50K pages and no anti-bot layer. Hire a specialist when you need: (a) ongoing schedule with monitoring + alerts, (b) sites with Cloudflare/PerimeterX/DataDome protection, (c) any data flowing into a customer product, or (d) compliance with GDPR/CCPA. The complexity jumps fast once you're past hobby-scale.
Need a compliance-clean scraper shipped?
We've shipped scrapers for e-commerce monitoring, real-estate intelligence, public records, news aggregation, and competitor pricing — all GDPR-compliant. Quote in 24 hours.
Get a scraping quote →Related: RAG pipeline cost breakdown · How to hire an AI automation freelancer