scraper-builder

name scraper-builder description Build production-ready web scrapers for any website using Bright Data infrastructure. Guides you through site analysis, API selection, selector extraction, pagination handling, and complete scraper implementation. Use this skill whenever the user wants to build a scraper, create a crawler, extract data from a website, scrape product pages, handle pagination, build a data pipeline from a web source, or automate data collection from any site — even if they don't explicitly say 'scraper'. Triggers on phrases like 'build a scraper for', 'scrape data from', 'extract products from', 'crawl pages on', 'get data from [website]', or 'I need to pull data from'. Scraper Builder You are building a production-ready web scraper for the user. Your job is to guide them from "I want data from site X" to a working, robust scraper that handles real-world challenges like pagination, dynamic content, anti-bot protection, and data parsing. Critical: Always Validate Your Output After building the scraper, always run it on a small sample (1-3 pages) and show the extracted data to the user before scaling up. If the output is empty, malformed, or missing fields, iterate — fix selectors, switch APIs, or adjust the parsing logic. A scraper that doesn't produce clean data is not done. Take your time with the reconnaissance phase. Spending 2 minutes analyzing the HTML upfront prevents hours of debugging later. Quality is more important than speed here. How This Skill Works This skill orchestrates Bright Data's four APIs to build scrapers intelligently. Rather than writing fragile custom scraping code, you analyze the target site first, then pick the most reliable and cost-effective extraction method. The decision tree is: Does a pre-built scraper already exist? → Use Web Scraper API (zero parsing code needed) Is the page static / no interaction needed? → Use Web Unlocker API (cheapest, simplest) Does the page need clicks, scrolls, or JS interaction? → Use Browser API (full automation) Need search engine results? → Use SERP API The skill produces complete, runnable code — not pseudocode or outlines. Phase 1: Understand the Target Before writing any code, you need to understand what the user wants and what the site looks like. Ask these questions (skip any the user already answered): What site? — The target URL or domain What data? — Which fields they need (product names, prices, reviews, etc.) What scope? — Single page, category pages, search results, entire site section? Pagination? — Do they need to scrape across multiple pages? Volume? — Roughly how many items/pages? (affects sync vs async choice and concurrency strategy — see references/concurrency-guide.md ) Output format? — JSON, CSV, database? (default to JSON if unspecified) Language preference? — Python or Node.js? (default to Python if unspecified) Don't over-interview. If the user says "build a scraper for Amazon product pages", you already know: site=Amazon, data=product details, scope=product pages. Jump ahead. Phase 2: Check for Pre-Built Scrapers Before doing any custom work, check if Bright Data already has a scraper for this domain. This is the fastest, cheapest, and most reliable path. Read references/supported-domains.md for the curated list of common pre-built scrapers. But the curated list may not be complete — Bright Data supports 100+ domains and adds new scrapers regularly. If you don't see the target domain in the curated list, query the live Dataset List API to check: curl -H "Authorization: Bearer $BRIGHTDATA_API_KEY " \ https://api.brightdata.com/datasets/list This returns every available scraper with its dataset_id and name. Search the results for the target domain. You can also browse the full documentation index at https://docs.brightdata.com/llms.txt to discover scraper-specific docs and supported parameters. If a pre-built scraper exists Use the Web Scraper API or Python SDK platform-specific scrapers. This gives you structured JSON with no parsing code needed. Python SDK approach (preferred): from brightdata import BrightDataClient async with BrightDataClient() as client: result = await client.scrape.amazon.products(url= "https://amazon.com/dp/B0CRMZHDG8" ) if result.success: print (result.data) # Structured product data REST API approach (shell/curl): bash scripts/datasets.sh amazon_product "https://www.amazon.com/dp/B09V3KXJPB" For bulk scraping with pre-built scrapers, use the async trigger/poll/fetch pattern: async with BrightDataClient() as client: # Trigger without waiting job = await client.scrape.amazon.products_trigger(url=url) # Poll until ready await job.wait(timeout= 180 , poll_interval= 10 , verbose= True ) # Fetch results data = await job.fetch() Skip to Phase 5 (pagination/orchestration) if the user needs multi-page scraping with a pre-built scraper. If no pre-built scraper exists Continue to Phase 3 — you need to analyze the site and build a custom scraper. Phase 3: Site Reconnaissance This is the critical step that separates reliable scrapers from brittle ones. You need to understand the site's structure before writing extraction code. Step 3a: Fetch the page HTML Use Web Unlocker to get the raw HTML. This tells you whether the content is server-rendered or client-rendered, and gives you the actual DOM to analyze. import requests import os API_KEY = os.environ[ "BRIGHTDATA_API_KEY" ] ZONE = os.environ[ "BRIGHTDATA_UNLOCKER_ZONE" ] response = requests.post( "https://api.brightdata.com/request" , headers={ "Authorization" : f"Bearer {API_KEY} " }, json={ "zone" : ZONE, "url" : "https://target-site.com/page" , "format" : "raw" } ) html = response.text Or use the scrape skill's shell script: bash skills/scrape/scripts/scrape.sh "https://target-site.com/page" Step 3b: Analyze the HTML structure Read references/site-analysis-guide.md for the detailed analysis playbook. Look at the fetched HTML and determine: Is the content in the HTML? If the data you need is present in the raw HTML, Web Unlocker is sufficient. If the HTML is mostly empty shells with JS framework markers ( <div id="root"></div> , <div id="__next"></div> , ng-app ), the content is client-rendered and you need Browser API. Identify reliable selectors. Find the CSS selectors or data attributes that target the data fields. Prefer selectors in this order (most reliable → least): data-* attributes (e.g., [data-testid="product-price"] ) — survive redesigns Semantic HTML with specific classes (e.g., .product-card .price ) id attributes — unique but may change Structural selectors (e.g., div > span:nth-child(2) ) — fragile, avoid Identify the data pattern. Is it: List page — multiple items in a repeating structure (product grid, search results) Detail page — single item with many fields (product page, profile) Paginated — multiple pages of results with next/prev controls Infinite scroll — content loads on scroll (needs Browser API) API-backed — check the Network tab pattern; some sites fetch data from JSON APIs Check for hidden APIs. Many modern sites load data via XHR/fetch calls to internal APIs. If you see structured JSON endpoints in the page source or network activity, hitting those directly through Web Unlocker is often cleaner than parsing HTML. Step 3c: Decide the extraction approach Based on your analysis: Finding Approach Content in HTML, no interaction needed Web Unlocker — fetch HTML, parse with BeautifulSoup/Cheerio Content loaded via JSON API Web Unlocker — hit the API endpoint directly Content requires JS rendering Browser API — render then extract Content needs click/scroll/interaction Browser API — automate the interaction Infinite scroll pagination Browser API — scroll and collect Standard URL-based pagination Web Unlocker — iterate page URLs CAPTCHA-heavy site Browser API — auto-solves CAPTCHAs Phase 4: Build the Extractor Now write the actual extraction code. The approach depends on Phase 3's decision. Approach A: Web Unlocker + HTML Parsing Best for static sites or sites with server-rendered HTML. This is the cheapest and fastest approach. import requests import os from bs4 import BeautifulSoup API_KEY = os.environ[ "BRIGHTDATA_API_KEY" ] ZONE = os.environ[ "BRIGHTDATA_UNLOCKER_ZONE" ] def fetch_page ( url: str ) -> str : """Fetch a page through Bright Data Web Unlocker.""" response = requests.post( "https://api.brightdata.com/request" , headers={ "Authorization" : f"Bearer {API_KEY} " }, json={ "zone" : ZONE, "url" : url, "format" : "raw" } ) response.raise_for_status() return response.text def parse_products ( html: str ) -> list [ dict ]: """Extract product data from HTML. Customize selectors per site.""" soup = BeautifulSoup(html, "html.parser" ) products = [] for card in soup.select( ".product-card" ): # Adjust selector product = { "name" : card.select_one( ".product-title" ).get_text(strip= True ), "price" : card.select_one( ".product-price" ).get_text(strip= True ), "url" : card.select_one( "a" )[ "href" ], # Add more fields as needed } products.append(product) return products # Usage html = fetch_page( "https://example.com/products" ) products = parse_products(html) Key patterns for robust parsing: Always use .get_text(strip=True) to clean whitespace Use .get("href", "") instead of ["href"] to avoid KeyError on missing attributes Wrap individual field extraction in try/except so one bad item doesn't kill the whole scrape Normalize prices (strip currency symbols, convert to float) in a separate step Approach B: Web Unlocker + Direct API Extraction When you discover the site loads data from a JSON API endpoint, hit it directly. This is the cleanest approach — no HTML parsing needed. import requests import json import os API_KEY = os.environ[ "BRIGHTDATA_API_KEY" ] ZONE = os.environ[ "BRIGHTDATA_UNLOCKER_ZONE" ] def fetch_api ( api_url: str ) -> dict : """Fetch a JSON API endpoint through Web Unlocker.""" response = requests.post( "https://api.brightdata.com/request" , headers={ "Authorization" : f"Bearer {API_KEY} " }, json={ "zone" : ZONE, "url" : api_url, "format" : "raw" } ) return json.loads(response.text) # Example: site with internal API data = fetch_api( "https://example.com/api/products?page=1&limit=50" ) products = data[ "results" ] # Already structured! Approach C: Browser API + Playwright Use when the site requires JavaScript rendering, interaction (clicks, scrolls, form fills), or has aggressive anti-bot measures. import asyncio from playwright.async_api import async_playwright AUTH = os.environ.get( "BROWSER_AUTH" , "brd-customer-CUSTOMER_ID-zone-ZONE_NAME:PASSWORD" ) async def scrape_with_browser ( url: str ) -> str : """Scrape a page using Bright Data Browser API.""" async with async_playwright() as p: browser = await p.chromium.connect_over_cdp( f"wss:// {AUTH} @brd.superproxy.io:9222" ) page = await browser.new_page() page.set_default_navigation_timeout( 120_000 ) # 2 minutes — required # Block unnecessary resources to reduce bandwidth costs await page.route( "**/*.{png,jpg,jpeg,gif,svg,css,woff,woff2}" , lambda route: route.abort()) await page.goto(url, wait_until= "domcontentloaded" ) # Wait for the content you need to appear await page.wait_for_selector( ".product-card" , timeout= 30_000 ) # Extract data using page.evaluate for performance products = await page.evaluate( """ () => Array.from(document.querySelectorAll('.product-card')).map(card => ({ name: card.querySelector('.product-title')?.textContent?.trim(), price: card.querySelector('.product-price')?.textContent?.trim(), url: card.querySelector('a')?.href, })) """ ) await browser.close() return products Browser API rules you must follow: Always set navigation timeout to 120 seconds ( set_default_navigation_timeout(120_000) ) One page.goto() per session — for a new URL, create a new browser connection Use wait_until="domcontentloaded" not networkidle (SPAs never reach networkidle) Wait for specific selectors rather than arbitrary delays Block images, CSS, fonts to reduce bandwidth costs Use page.evaluate() for bulk extraction — it's faster than individual selector calls Approach D: Browser API for Infinite Scroll For sites that load more content when you scroll down. async def scrape_infinite_scroll ( url: str , max_items: int = 100 ) -> list : """Scrape a page with infinite scroll.""" async with async_playwright() as p: browser = await p.chromium.connect_over_cdp( f"wss:// {AUTH} @brd.superproxy.io:9222" ) page = await browser.new_page() page.set_default_navigation_timeout( 120_000 ) await page.route( "**/*.{png,jpg,jpeg,gif,svg,woff,woff2}" , lambda route: route.abort()) await page.goto(url, wait_until= "domcontentloaded" ) all_items = [] previous_count = 0 while len (all_items) < max_items: # Scroll to bottom await page.evaluate( "window.scrollTo(0, document.body.scrollHeight)" ) await page.wait_for_timeout( 2000 ) # Wait for content to load # Extract all currently visible items items = await page.evaluate( """ () => Array.from(document.querySelectorAll('.item-selector')).map(el => ({ // ... extract fields })) """ ) all_items = items if len (all_items) == previous_count: break # No new content loaded — we've reached the end previous_count = len (all_items) await browser.close() return all_items[:max_items] Phase 5: Handle Pagination Most scraping tasks involve multiple pages. The approach depends on the pagination type. Read references/pagination-patterns.md for detailed pagination strategies. Pattern 1: URL-Based Pagination (most common) Pages are accessed via URL parameters like ?page=2 or ?offset=20 . import time def scrape_all_pages ( base_url: str , max_pages: int = 50 ) -> list [ dict ]: """Scrape all pages of a paginated listing.""" all_items = [] for page_num in range ( 1 , max_pages + 1 ): url = f" {base_url} ?page= {page_num} " html = fetch_page(url) items = parse_products(html) if not items: break # No more results all_items.extend(items)