Brightdata SDK

Bright Data SDK

Access web data through a unified Python SDK. One client, eight service categories: platform scraping, platform search, web search (SERP), AI-powered discovery, datasets, web unlocking, browser automation, and scraper studio.

Always use the client as a context manager. In synchronous environments (scripts, notebooks, Claude Code), use SyncBrightDataClient. In async environments, use BrightDataClient. Both use the same method names — the sync client wraps calls automatically. Note: the sync client currently has limited platform coverage — see the sync compatibility note in references/scrapers.md for details. For unsupported platforms or the datasets API, use the async client (BrightDataClient).

Service Selection (decide first, then look up the specific method)

Use this decision tree to pick the right service BEFORE reaching for any specific method. Most routing failures come from skipping this step and pattern-matching on user keywords instead.

Have a URL?
  ├── On a supported platform (Amazon, LinkedIn, Facebook, Instagram, YouTube,
  │   TikTok, Reddit, ChatGPT, Perplexity, Pinterest, DigiKey)?
  │     → Platform scraping: client.scrape.<platform>.<method>(url=...)
  │
  ├── Generic page (not on any supported platform)?
  │     → Web unlocker: client.scrape_url(url=...)
  │
  └── Need login / JavaScript / click-scroll-fill / CAPTCHA / multi-step navigation?
        → Browser API: client.browser.get_connect_url() (then connect with Playwright)

No URL?
  ├── Want entities matching natural-language criteria
  │   ("find AI startups in Berlin", "competitors of Acme Corp", "people who worked at X")?
  │     → Discover: client.discover(query=..., intent=...)
  │
  ├── Want web pages / articles / search-result links
  │   ("search Google for X", "find pages about Y")?
  │     → SERP: client.search.google(query=...) [or .bing / .yandex]
  │
  ├── Want to search WITHIN a specific platform
  │   ("find products on Amazon", "TikTok videos by hashtag", "Pinterest pins about recipes")?
  │     → Platform search: client.search.<platform>.<method>(...)
  │
  └── Want bulk historical data at scale
      ("all LinkedIn companies in tech", "historical Amazon prices", "every Zillow listing in Texas")?
        → Datasets: client.datasets.<name>(filter=...) (then .download(snapshot_id))

Edge cases:

• URL on supported platform BUT user explicitly mentions login/click/scroll/JS → Browser API (the interaction trumps the platform).
• URL on supported platform BUT scraper returns 403/blocked → fall back to client.scrape_url() (web unlocker).
• "Find/research/who are X" with a URL alongside (e.g., "find competitors of acme.com") → still Discover; the URL is context, not the scrape target.

Method Names: Verify Before Asserting

Before claiming any platform method exists, doesn't exist, or asserting a platform isn't supported, you MUST consult references/scrapers.md first. Past evals show the model has hallucinated method names (e.g., client.scrape.linkedin.people — does NOT exist; use .profiles) and falsely claimed platforms unsupported (e.g., Pinterest is supported via both client.scrape.pinterest and client.search.pinterest).

Rule: load references/scrapers.md before naming any specific platform method. The reference file lists every platform, every method signature, and the sync/async availability matrix. Verify, don't assume. If you've already loaded references/scrapers.md in this session, consult what's in context — no need to reload.

Known hallucinations (these names do NOT exist in the SDK — the model has invented them in past evals):

This list grows as new hallucinations are observed in evals. If you're tempted to write a method name that "feels right" but you haven't seen in references/scrapers.md, treat it as a likely hallucination — load the reference and verify.

Useful Standalone Methods

Methods that don't belong to any specific workflow — easy to overlook because they're not tied to a platform or a routing decision. The model has hallucinated some of these (client.list_datasets() instead of client.datasets.list()); use the canonical names below.

How to Handle Requests

Exploring capabilities

If the user wants to know what's available or asks "what can this do?", describe these 8 categories. Each follows the template: Name — what it does — example invocation — when to use.

1. Platform scraping — extract structured data from 11 supported platforms by URL. client.scrape.<platform>.<method>(url=...) (platforms: Amazon, LinkedIn, Facebook, Instagram, YouTube, TikTok, Reddit, ChatGPT, Perplexity, Pinterest, DigiKey). Use when the user has a URL on a supported platform and wants structured fields (price, profile data, post engagement, etc.).

2. Platform search — search within a specific platform by keyword/profile/filter. client.search.<platform>.<method>(...). Use for "find products on Amazon by keyword", "discover TikTok videos by hashtag", "find Pinterest pins about recipes" — i.e., the user wants to search WITHIN a platform but doesn't have a specific URL.

3. Web search (SERP) — get structured search engine results (titles, links, snippets, rankings). client.search.google(query=...) (or .bing / .yandex). Use for "search Google for X", "find pages/articles about Y", "look up news on Z" — i.e., the user wants web pages, not entities.

4. Discover (AI-powered) — client.discover(query=..., intent=...) to find entities (companies, people, products, places) matching natural-language criteria. Use for "find AI startups in Berlin", "competitors of Acme Corp", "people who worked at Stripe", "research the SaaS pricing landscape" — i.e., the user wants a list of entities matching a description, not web pages.

5. Datasets — access 310+ pre-built datasets with historical/bulk data at scale. client.datasets.<name>(filter=...) (then .download(snapshot_id)). Use for "bulk LinkedIn company data", "historical Amazon prices for electronics", "all Zillow listings in Texas" — i.e., the user wants many records at once, not live data on one page.

6. Web unlocker — scrape any URL with anti-bot bypass for sites without a dedicated platform scraper. client.scrape_url(url=...). Use when the URL is on a generic website (no dedicated scraper) or as fallback when a platform scraper returns 403/blocked.

7. Browser API — connect to a remote browser via CDP (Chrome DevTools Protocol) for real-browser interaction. client.browser.get_connect_url() (then connect with Playwright/Puppeteer). Use for login flows, JavaScript-heavy single-page apps, click/scroll/fill interactions, CAPTCHA — i.e., anything requiring a real browser session. Most expensive option; use only when simpler methods can't accomplish the task.

8. Scraper Studio — run pre-built or custom scraping templates configured in the Bright Data dashboard. client.scraper_studio.run(collector="c_xxx", input={...}). Use when the user provides a collector ID for a template not covered by platform scrapers.

Offer to load the relevant reference file for details on any category.

Data extraction from a specific URL

The user has a URL and wants structured data from it.

If the URL is from a supported platform (Amazon, LinkedIn, Facebook, Instagram, YouTube, TikTok, Reddit, ChatGPT, Perplexity, Pinterest, DigiKey — see references/scrapers.md for the full list and available methods):

• Use client.scrape.<platform>.<method>(url=...)
• Read references/scrapers.md for available methods per platform

If the URL is from an unsupported platform or a generic website:

• Use client.scrape_url(url=...) for raw page data with anti-bot bypass
• Read references/advanced.md for web unlocker options

If the user has MULTIPLE URLs (batch):

• Use BrightDataClient and trigger methods (_trigger suffix) to avoid sequential blocking
• Fire all triggers first, then collect results with job.wait() and job.to_result()
• Read references/advanced.md for batch execution patterns

Research or discovery without a specific URL

The user wants to find information but doesn't have a starting URL.

For web search results (links, snippets, rankings):

• Use client.search.google(query=...), client.search.bing(query=...), or client.search.yandex(query=...)
• Read references/search.md for available search engines and parameters

For platform-specific search (find products on Amazon, profiles on LinkedIn, videos on YouTube, etc.):

• Use client.search.<platform>.<method>(...)
• Read references/scrapers.md — search methods are listed under each platform

For deeper discovery (find companies, people, or entities matching criteria):

• Use client.discover(query=..., intent=...)
• The Discover API requires an intent phrase, not just keywords
• Read references/search.md for discover API details

Bulk or historical data needs

The user asks for "bulk data", "historical data", "database", "list of", or wants data at scale without scraping individual pages.

• Use client.datasets.list() at runtime to discover available datasets
• Read references/datasets-overview.md for dataset categories and usage patterns
• Create a filtered snapshot: snapshot_id = client.datasets.<name>(filter={...})
• Download data: data = client.datasets.<name>.download(snapshot_id) (default format is jsonl; also supports json, csv)
• Snapshots take time to build — download blocks until ready (up to 5 minutes)

Multi-step research workflow

The user has a broad research goal (e.g., "research competitors in Berlin").

Step 1: Find sources

• client.discover(query=..., intent=...) for entity-level discovery
• OR client.search.google(query=...) for web search results

Step 2: Extract data from discovered sources

• client.scrape.<platform>.<method>(url=...) on each discovered URL
• Use trigger methods for batch processing if many URLs

Step 3: Optionally enrich with bulk data

• Check client.datasets for historical context on the entities found

Interactive web tasks

The user needs login, clicking, scrolling, form filling, or JavaScript execution.

• Use client.browser.get_connect_url() to get a CDP WebSocket URL
• Connect with Playwright, Puppeteer, or another CDP client
• This is the most expensive option — only use when simpler methods cannot accomplish the task
• Read references/advanced.md for browser API details

Scraper Studio templates

The user wants to use a pre-built or custom scraping template.

• Use client.scraper_studio.run(collector="c_xxx", input={...})
• Requires a collector ID — the user must provide this or know which template to use
• Read references/advanced.md for scraper studio details

Gotchas