Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems.
$ npx skills add databricks/databricks-agent-skills --skill spark-python-data-source -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install databricks/databricks-agent-skills spark-python-data-source --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/databricks/databricks-agent-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/experimental/spark-python-data-source .claude/skills/spark-python-data-source && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "spark-python-data-source" agent skill from https://github.com/databricks/databricks-agent-skills/tree/main/experimental/spark-python-data-source into .claude/skills/spark-python-data-source/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "spark-python-data-source", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/databricks/databricks-agent-skills/tree/main/experimental/spark-python-data-sourceType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add databricks/databricks-agent-skills --skill spark-python-data-source -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install databricks/databricks-agent-skills spark-python-data-source --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/databricks/databricks-agent-skills.git skills-src && mkdir -p .agents/skills && cp -r skills-src/experimental/spark-python-data-source .agents/skills/spark-python-data-source && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "spark-python-data-source" agent skill from https://github.com/databricks/databricks-agent-skills/tree/main/experimental/spark-python-data-source into .agents/skills/spark-python-data-source/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "spark-python-data-source", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add databricks/databricks-agent-skills --skill spark-python-data-source -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install databricks/databricks-agent-skills spark-python-data-source --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/databricks/databricks-agent-skills.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/experimental/spark-python-data-source .cursor/skills/spark-python-data-source && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "spark-python-data-source" agent skill from https://github.com/databricks/databricks-agent-skills/tree/main/experimental/spark-python-data-source into .cursor/skills/spark-python-data-source/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "spark-python-data-source", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/databricks/databricks-agent-skills.git --path experimental/spark-python-data-source--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add databricks/databricks-agent-skills --skill spark-python-data-source -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install databricks/databricks-agent-skills spark-python-data-source --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/databricks/databricks-agent-skills.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/experimental/spark-python-data-source .gemini/skills/spark-python-data-source && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "spark-python-data-source" agent skill from https://github.com/databricks/databricks-agent-skills/tree/main/experimental/spark-python-data-source into .gemini/skills/spark-python-data-source/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "spark-python-data-source", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install databricks/databricks-agent-skills spark-python-data-sourceInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add databricks/databricks-agent-skills --skill spark-python-data-source -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/databricks/databricks-agent-skills.git skills-src && mkdir -p .github/skills && cp -r skills-src/experimental/spark-python-data-source .github/skills/spark-python-data-source && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "spark-python-data-source" agent skill from https://github.com/databricks/databricks-agent-skills/tree/main/experimental/spark-python-data-source into .github/skills/spark-python-data-source/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "spark-python-data-source", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add databricks/databricks-agent-skills --skill spark-python-data-source -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install databricks/databricks-agent-skills spark-python-data-source --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/databricks/databricks-agent-skills.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/experimental/spark-python-data-source .opencode/skills/spark-python-data-source && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "spark-python-data-source" agent skill from https://github.com/databricks/databricks-agent-skills/tree/main/experimental/spark-python-data-source into .opencode/skills/spark-python-data-source/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "spark-python-data-source", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
spark-python-data-sourceBuild custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems.
Spark Python Data Source is an agent skill from databricks/databricks-agent-skills, published by the product's own GitHub organization. Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or…
Its SKILL.md is about 2.1k tokens, which your agent loads only when the skill is triggered. The skill folder holds 14 other files, including reference files and assets (for example `agents/openai.yaml`, `references/authentication-patterns.md` and `references/error-handling.md`). Compatibility notes: Requires databricks CLI (= v1.0.0)
It sits in Data & Analytics, covering Event-driven systems and DataFrames. It works with Python and Apache Spark. The repository describes itself as: Databricks AI Tools: skills and plugins for building on Databricks with Claude Code, Cursor, Codex, GitHub Copilot, and other AI coding agents.
4 steps, taken from the first numbered list in SKILL.md.
Read from SKILL.md and the folder at commit f4fcec5. It shows what the files ask for, not the result of running them.
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Shell commands in SKILL.md call:
uvFrom the folder's file list and the shell code blocks in SKILL.md.
Links to these hosts (documentation or services it may open):
github.comdocs.databricks.comspark.apache.orgFrom URLs in SKILL.md, links to its own repository left out.
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Requires databricks CLI (>= v1.0.0)
From compatibility in the SKILL.md frontmatter.
Spark Python Data Source loads about 2.1k tokens when it runs, and up to ~24k if it reads all its reference files. Until then it costs about 154 tokens; SKILL.md has 607 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
Its licence (Custom licence) doesn't allow us to republish the file, so here is its outline and opening line. It has 607 words (~2,068 tokens).
“Build custom Python data sources for Apache Spark 4.0+ to read from and write to external systems in batch and streaming modes.”
SKILL.md and 11 other files (references, assets) in experimental/spark-python-data-source of databricks/databricks-agent-skills.
Open the folder on GitHubat commit f4fcec5
Spark Python Data Source next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Spark Python Data Source this skilldatabricks/databricks-agent-skills | 345 | — | ~2.1k | Automated safety check: Pass | Custom licence | |
| Flowfile Frame And CodegenEdwardvaneechoud/Flowfile | 370 | — | ~12k | Automated safety check: Pass | MIT | |
| Datafusion Pythonapache/datafusion-python | 606 | — | ~7.8k | Automated safety check: Pass | Apache-2.0 | |
| Neo4j Spark Skillneo4j-contrib/neo4j-skills | 114 | — | ~4.1k | Automated safety check: Notes | MIT | |
| Spark EngineerFerroxLabs/wayland | 608 | — | ~4.2k | Automated safety check: Pass | Apache-2.0 | |
| Polar Python SDKpolarsource/polar | 10k | — | ~1.8k | Automated safety check: Pass | Apache-2.0 |
Edwardvaneechoud/Flowfile
Deep dive into flowfileframe — the Polars-LazyFrame-shaped Python API that builds an in-process flowfilecore FlowGraph as a side effect of every method call — covering the FlowFrame/Expr internals…
apache/datafusion-python
A skill your agent uses when the user is writing datafusion-python (Apache DataFusion Python bindings) DataFrame or SQL code.
neo4j-contrib/neo4j-skills
A skill your agent uses when reading from or writing to Neo4j with Apache Spark or Databricks using the Neo4j Connector for Apache Spark 6.0 (org.neo4j.connectors:spark) or 5.x…
FerroxLabs/wayland
Apache Spark expertise covering RDD vs DataFrame vs Dataset APIs, partitioning strategies, shuffle optimization, broadcast joins, caching, Spark SQL, structured streaming, UDFs, cluster sizing…
polarsource/polar
Integrate Polar billing in server-side Python applications using the versioned Polar and PolarAsync clients.
wshobson/agents
Speed up slow Apache Spark jobs by tuning partitions, shuffles, data skew, caching and executor memory, with do and don't rules for PySpark code.
databricks/databricks-agent-skills
Databricks SQL (DBSQL) advanced features and SQL warehouse capabilities.
databricks/databricks-agent-skills
Generate realistic synthetic data using Spark + Faker (strongly recommended).
databricks/databricks-agent-skills
Databricks Model Serving endpoint lifecycle and ops. An agent skill from databricks/databricks-agent-skills.
databricks/databricks-agent-skills
Databricks development guidance including Python SDK, Databricks Connect, CLI, and REST API.
databricks/databricks-agent-skills
Comprehensive guide to Spark Structured Streaming for production workloads.
databricks/databricks-agent-skills
Design the UX of custom-code Databricks Apps (AppKit/React) data screens — KPI/overview pages, reports, charts, tables, and Genie/chat data assistants — mapped to concrete AppKit components.
Works with
Categories
Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Spark Python Data Source is an agent skill from databricks/databricks-agent-skills, published by the product's own GitHub organization. Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems.
Spark Python Data Source fits situations like: someone wants to connect Spark to an external system (database; custom protocol); build a Spark connector; plugin in Python.
Run `npx skills add databricks/databricks-agent-skills --skill spark-python-data-source -a claude-code`. Or copy the skill folder (experimental/spark-python-data-source in databricks/databricks-agent-skills) into .claude/skills/spark-python-data-source in your project. Claude Code loads it when a task matches its description.
Run `npx skills add databricks/databricks-agent-skills --skill spark-python-data-source -a codex`. Or copy the skill folder (experimental/spark-python-data-source in databricks/databricks-agent-skills) into .agents/skills/spark-python-data-source in your project. Codex loads it when a task matches its description.
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add databricks/databricks-agent-skills --skill spark-python-data-source -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/spark-python-data-source, .gemini/skills/spark-python-data-source, .github/skills/spark-python-data-source and .opencode/skills/spark-python-data-source in your project.
Going by SKILL.md and its folder, Spark Python Data Source needs the command-line tools its instructions call (uv). Our summary lists: Python 3. Compatibility (from SKILL.md): Requires databricks CLI (>= v1.0.0).
SKILL.md names 3 domains. As links in the text: github.com, docs.databricks.com and spark.apache.org. This is read from the text; nothing was executed.
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
Spark Python Data Source has a licence file (the repository's licence) that doesn't match a standard licence. Read it on GitHub before reusing the skill.
About 2.1k tokens (SKILL.md is roughly 8.3k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 21k tokens, read only when the agent opens those files.
Skills that share tags, products or a category with Spark Python Data Source: Flowfile Frame And Codegen (Edwardvaneechoud/Flowfile, 370 stars), Datafusion Python (apache/datafusion-python, 606 stars), Neo4j Spark Skill (neo4j-contrib/neo4j-skills, 114 stars) and Spark Engineer (FerroxLabs/wayland, 608 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
databricks (a GitHub organization, an official publisher) maintains it in databricks/databricks-agent-skills, which has 345 GitHub stars. The repository holds 32 skills in this directory. The repository was last updated on October 6, 2026.
Source: databricks/databricks-agent-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.
