RAG Pipeline

What it does

When your agent uses it

• The user wants to build a RAG pipeline
• Add web search to my LLM/agent
• Ground my model in live web data
• Build a search engine over the web

Example prompts

• “build a RAG pipeline”
• “add web search to my LLM/agent”
• “ground my model in live web data”
• “/rag-pipeline”

Requirements

• Python 3
• A credential in BRIGHTDATA_API_TOKEN

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

1. 1Retrieval returns non-empty, on-topic chunks for a known test query (eyeball top-k links).
2. 2No block-page / empty content made it into the index — spot-check stored chunks.
3. 3Citations resolve — every [n] the LLM emits maps to a real source link in the retrieved set.
4. 4Freshness is honored — if the app promises current data, confirm live retrieval (or a recent re-ingest), not a stale index.
5. 5Grounding check — answers are supported by retrieved content, not the model's prior; test with a question whose answer only exists in a…

What it can do on your machine

Read from SKILL.md and the folder at commit 81f51af. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  No scripts in the folder and no shell commands in SKILL.md (its code samples are javascript).

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names these keys or tokens, usually read from environment variables:

  • BRIGHTDATA_API_TOKEN

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

RAG Pipeline loads about 1.9k tokens when it runs, and up to ~4.4k if it reads all its reference files. Until then it costs about 195 tokens; SKILL.md has 640 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~195

When it runs · the whole SKILL.md, loaded when a task matches

~1.9k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~4.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from brightdata/skills at commit 81f51af, republished under its MIT licence (© brightdata). 640 words, ~1,850 tokens.

name

rag-pipeline

description

Build a RAG (retrieval-augmented generation) pipeline or a custom search engine on top of Bright Data's Discover API — using intent-ranked web results + parsed page content as the retrieval/ingestion layer for an LLM or vector store. Use when the user wants to "build a RAG pipeline", "add web search to my LLM/agent", "ground my model in live web data", "build a search engine over the web", "ingest web content into a vector DB / knowledge base", or "give my chatbot retrieval". Covers both live retrieval (Discover at query time as a web-grounded retriever) and ingestion (Discover → chunk → embed → vector store → retrieve). Built on the `discover-api` skill. For a one-off written report use `live-research`; for raw markdown of specific known URLs use `scrape`.

metadata.author

Bright Data

metadata.version

1.0

Does the corpus change every query, or is it a stable knowledge base?

  ├── Per-query, always-fresh ("ground each answer in live web data")
  │     → LIVE RETRIEVAL: Discover(include_content) at query time → top-k → LLM
  │       Pros: always current, no storage. Cons: per-query latency + cost.
  │
  └── Reused across many queries ("build a knowledge base / search engine")
        → INGESTION: Discover(include_content) → chunk → embed → vector store
          then at query time: embed query → vector search → (rerank) → LLM
          Pros: fast queries, cacheable. Cons: can go stale (re-ingest on a schedule).

import { bdclient } from '@brightdata/sdk';
const client = new bdclient(); // BRIGHTDATA_API_TOKEN

async function retrieve(question, k = 6) {
  const res = await client.discover(question, {
    intent: `authoritative sources that directly answer: ${question}`,
    includeContent: true,
    numResults: Math.min(k * 2, 20),  // over-fetch, then trim
  });
  // NOTE: the JS SDK returns a WRAPPER object, not a bare array:
  //   { success, data: [ {link,title,description,relevance_score,content?} ], totalResults, cost, taskId, ... }
  // The result rows are in `.data` (CLI/REST use `.results` instead — see discover-api).
  if (!res.success) throw new Error(`discover failed: ${res.error ?? 'unknown'}`);
  return (res.data ?? [])
    .filter(r => r.content && !/just a moment|captcha|access denied|not found/i.test(r.content) && r.content.length > 200)
    .sort((a, b) => b.relevance_score - a.relevance_score)
    .slice(0, k);
}
// → build a prompt from sources[].content, ask the LLM to answer WITH [n] citations to sources[].link

© brightdata, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 1 other file (references) in skills/rag-pipeline of brightdata/skills.

• SKILL.md
• references/code.md

Open the folder on GitHubat commit 81f51af