New: 97,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Executing Spark

by data-goblin in data-goblin/power-bi-agentic-development

Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access.

GPL-3.0Auto-check passedData & Analytics

Install Executing Spark

skills CLI
$ npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install data-goblin/power-bi-agentic-development executing-spark --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/data-goblin/power-bi-agentic-development.git skills-src && mkdir -p .claude/skills && cp -r skills-src/plugins/etl/skills/executing-spark .claude/skills/executing-spark && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
executing-spark
GitHub stars
1k
Token cost
~1.7k tokens
SKILL.md length
699 words
Files
3 (incl. references)
Skills in repo
33
Repo updated
First seen
Licence
GPL-3.0

At a glance

Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access.

  • Asks to run PySpark in Fabric
  • SKILL.md covers Prerequisites, Critical: Authentication, Lifecycle and Getting IDs, plus 9 more sections
  • Calls az; reaches api.fabric.microsoft.com
  • Create a Livy session

What it does

Executing Spark is an agent skill from data-goblin/power-bi-agentic-development. Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Automatically invoke when the user asks to "run PySpark in Fabric", "create a Livy session", "execute Python on Fabric compute", "run Spark without a notebook", "submit code to Fabric", "ephemeral Spark execution", "run ETL in Fabric".

Its SKILL.md is about 1.7k tokens, which your agent loads only when the skill is triggered. The skill folder holds 3 other files, including reference files (for example `references/example-script.md` and `references/livy-api.md`).

It sits in Data & Analytics, covering Data pipelines and ETL. It works with Apache Spark and Python. The repository describes itself as: Power BI AI skills and Power BI agents for Claude Code and GitHub Copilot: a plugin marketplace of Power BI skills, subagents, and hooks for semantic models, DAX, TMDL, reports… The licence is GPL-3.0.

When your agent uses it

  • Asks to run PySpark in Fabric
  • Create a Livy session
  • Execute Python on Fabric compute
  • Run Spark without a notebook

Example prompts

  • “run PySpark in Fabric”
  • “create a Livy session”
  • “execute Python on Fabric compute”
  • “/executing-spark”

Requirements

  • Python 3

What it can do on your machine

Read from SKILL.md and the folder at commit 41886f2. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • az

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Hosts in commands or code, which the agent is likely to contact:

    • api.fabric.microsoft.com

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Executing Spark loads about 1.7k tokens when it runs, and up to ~3.8k if it reads all its reference files. Until then it costs about 102 tokens; SKILL.md has 699 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~102
When it runs · the whole SKILL.md, loaded when a task matches
~1.7k
With references · SKILL.md plus every file in references/, read only if the agent opens them
~3.8k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from data-goblin/power-bi-agentic-development at commit 41886f2, republished under its GPL-3.0 licence (© data-goblin). 699 words, ~1,662 tokens.

Download SKILL.mdSave it as .claude/skills/executing-spark/SKILL.md (or your agent's skills folder). This skill also uses 2 other files; get the full folder from GitHub.
name
executing-spark
description
Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Automatically invoke when the user asks to "run PySpark in Fabric", "create a Livy session", "execute Python on Fabric compute", "run Spark without a notebook", "submit code to Fabric", "ephemeral Spark execution", "run ETL in Fabric".

Executing Spark Code in Fabric (No Notebook)

Run arbitrary PySpark or Python code on Fabric Spark compute via the Livy API. No notebook artifact is created or persisted; sessions are ephemeral. Full read/write access to lakehouse Delta tables via Spark SQL.

Prerequisites

  • Azure CLI authenticated (az login)
  • A lakehouse in the target workspace (the Livy session runs against it)
  • Fabric capacity (F or trial)

Critical: Authentication

The Livy API requires a token from az account get-access-token --resource https://api.fabric.microsoft.com. Tokens from fab auth do not work for OneLake storage access inside the Spark session.

python
import subprocess, json

result = subprocess.run(
    ["az", "account", "get-access-token", "--resource", "https://api.fabric.microsoft.com"],
    capture_output=True, text=True
)
token = json.loads(result.stdout)["accessToken"]

Do not output or log the token. Pass it directly to the API call.

Lifecycle

1. Create session   POST .../sessions              {"kind": "pyspark"}
2. Wait for idle    GET  .../sessions/{id}          poll until state: "idle" (~30-90s)
3. Submit code      POST .../sessions/{id}/statements   {"code": "...", "kind": "pyspark"}
4. Get result       GET  .../sessions/{id}/statements/{n}   poll until state: "available"
5. Delete session   DELETE .../sessions/{id}        ALWAYS do this

Base URL: https://api.fabric.microsoft.com/v1/workspaces/{wsId}/lakehouses/{lhId}/livyapi/versions/2023-12-01

CRITICAL: Always delete sessions when done. Idle sessions consume Fabric capacity units (CUs). A forgotten session burns compute until it times out (default: 20 minutes). In automation, wrap cleanup in a finally block.

Getting IDs

bash
WS_ID=$(fab get "Workspace.Workspace" -q "id" | tr -d '"')
LH_ID=$(fab get "Workspace.Workspace/Lakehouse.Lakehouse" -q "id" | tr -d '"')

Submitting Code

Submit PySpark or pure Python as statements. The spark object is available automatically.

python
# Statement payload
{"code": "df = spark.sql('SELECT * FROM products LIMIT 10')\ndf.show()", "kind": "pyspark"}

Results are in output.data["text/plain"] when state: "available" and output.status: "ok".

What Works

  • spark.sql("SELECT ...") ; full Spark SQL against lakehouse tables
  • spark.sql("SHOW TABLES") ; metastore access
  • df.write.mode("overwrite").saveAsTable(...) ; write Delta tables
  • Pure Python (pandas, numpy, pyarrow); runs on Spark container
  • In-memory Spark DataFrames and transformations
  • Multiple sequential statements in one session

What Does Not Work

  • deltalake (delta-rs) is not pre-installed; use Spark SQL instead
  • notebookutils has limited functionality (no FUSE mount at /lakehouse/default/)
  • Tokens from fab auth ; must use az CLI token
  • Tokens expire after ~60 minutes; long sessions need token refresh

When to Use This vs Alternatives

ScenarioApproach
Quick read-only explorationDuckDB locally (fastest; see using-duckdb skill)
Write data back to lakehouseLivy session or notebook
Ephemeral transform; no artifactLivy session (this skill)
Complex multi-cell workflowNotebook (nb exec or portal)
Scheduled ETLNotebook via fab job run
Agent-driven compute (Dagster, orchestrators)Livy session

Persisting code as a notebook: poll the definition LRO tightly

This skill is for ephemeral execution with no artifact. When you instead want to persist or change a notebook (deploy new code, iterate on an existing one), that is an item-definition change, and the poll interval is the single biggest performance lever. fab import, nb create, and nb cell edit take 25-60s because they poll the create/update long-running operation at the server's advertised Retry-After: 20; the work itself finishes in ~1s, and neither CLI lets you change that interval. Poll the LRO at ~0.3s and the same deploy takes ~1-2s. The fabric-cli skill ships scripts/deploy_notebook.py which does this (auto-detects create vs update, --poll-interval default 0.3s); strongly prefer it over fab import / nb for any notebook definition change.

Show full SKILL.md (268 more words)Show less

Sessions vs Batch Jobs

A Livy session (this skill) is interactive: create it, submit statements, read output as it runs, delete it. It stays alive and you pay for idle time until you delete it or it times out (~20 min).

A Livy batch is one-shot: submit a single job (a file or inline job spec), poll it to a terminal state, done. No idle-CU footgun, nothing to remember to delete. For scheduled or fire-and-forget agent ETL, prefer a batch over a session; keep sessions for interactive, multi-statement work. Same base URL, /batches instead of /sessions -- see references/livy-api.md.

Livy vs Notebook Jobs: reading the outcome

A Livy statement returns its result directly in the response (output.status = ok/error), so you always know whether it worked. A notebook run via fab job run does not -- its job status reports Completed even when the notebook caught an exception and exited a failure payload. If you run notebooks as batch jobs instead of Livy, you must read the notebook's exit value to get its real verdict. The fabric-cli skill (in the fabric-cli plugin) documents that endpoint and ships scripts/run_notebook_checked.py for it.

References

  • references/livy-api.md -- Full API reference with endpoints (sessions + batches), request/response formats, and error handling
  • references/example-script.md -- Complete working script that creates a session, queries data, writes results, and cleans up
  • using-duckdb skill (same etl plugin) -- read-only Delta querying, local or in-notebook, when you don't need Spark compute
  • fabric-cli skill (fabric-cli plugin) -- nb exec / fab job run for notebooks, reading a notebook's exit value, the SQL-endpoint metadata sync after a Spark write, and scripts/deploy_notebook.py for fast notebook definition changes (tight LRO polling)

© data-goblin, GPL-3.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 2 other files (references) in plugins/etl/skills/executing-spark of data-goblin/power-bi-agentic-development.

  • SKILL.md
  • references/example-script.md
  • references/livy-api.md

Open the folder on GitHubat commit 41886f2

Compare with similar skills

Executing Spark next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Executing Spark compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Executing Spark this skilldata-goblin/power-bi-agentic-development1k—~1.7kAutomated safety check: PassGPL-3.0
Apache Spark Optimizationwshobson/agents40k8 repos~789Automated safety check: PassMIT
Spark EngineerFerroxLabs/wayland608—~4.2kAutomated safety check: PassApache-2.0
Crawl4AI Web Scrapingsmallnest/goclaw5981 repos~2.5kAutomated safety check: PassMIT
Migrate Glue Devendpoint To Interactive Sessionsaws-samples/aws-glue-samples1.5k—~3.6kAutomated safety check: PassMIT-0
Monitor With HaolemeHaolemeApp/Haoleme157—~1.3kAutomated safety check: PassAGPL-3.0

Similar skills

  • Apache Spark Optimization

    wshobson/agents

    Speed up slow Apache Spark jobs by tuning partitions, shuffles, data skew, caching and executor memory, with do and don't rules for PySpark code.

    40k GitHub starsUsed in 8 repos~789 tokens
    Data & AnalyticsAuto-check passed

  • Spark Engineer

    FerroxLabs/wayland

    Apache Spark expertise covering RDD vs DataFrame vs Dataset APIs, partitioning strategies, shuffle optimization, broadcast joins, caching, Spark SQL, structured streaming, UDFs, cluster sizing…

    608 GitHub stars~4.2k tokensUpdated yesterday
    Data & AnalyticsAuto-check passed

  • Crawl4AI Web Scraping

    smallnest/goclaw

    Scrapes sites, handles JavaScript-heavy pages and extracts structured data with Crawl4AI, through its crwl CLI or Python SDK, including schema-based extraction without an LLM.

    598 GitHub starsUsed in 1 repo~2.5k tokens
    Data & AnalyticsAuto-check passed

  • Migrate Glue Devendpoint To Interactive Sessions

    aws-samples/aws-glue-samples

    Official

    Migrate a legacy AWS Glue development endpoint to a Glue interactive session, following the official AWS migration checklist.

    1.5k GitHub stars~3.6k tokensUpdated 1 mo ago
    Data & AnalyticsAuto-check passed

  • Monitor With Haoleme

    HaolemeApp/Haoleme

    Selectively monitor important long-running or resource-intensive commands with Haoleme by prefixing them with hao, so status, output, and completion notifications sync to the mobile app.

    157 GitHub stars~1.3k tokensUpdated 1 mo ago
    Data & AnalyticsAuto-check passed

  • Tushare Plugin Builder

    Yourdaylight/stock_datasource

    Turns a Tushare API doc URL into a full data plugin for the stock_datasource repo: extractor, ClickHouse schema, query service, config and curl examples.

    188 GitHub stars~2.5k tokensUpdated 29 days ago
    Data & AnalyticsAuto-check passed

More from data-goblin/power-bi-agentic-development

All 33 skills in this repo

  • Paginated Report

    data-goblin/power-bi-agentic-development

    Author, validate, publish, and test Power BI paginated reports in the RDL format.

    1k GitHub stars~3.5k tokensUpdated 2 days ago
    Auto-check passed

  • Audit Tenant Settings

    data-goblin/power-bi-agentic-development

    Automatically invoke this skill whenever the user asks about Fabric tenant settings or Power BI tenant settings or auditing tenant settings.

    1k GitHub stars~4.7k tokensUpdated 2 days ago
    Auto-check passed

  • Bpa Rules

    data-goblin/power-bi-agentic-development

    Interactive BPA rule generation for Power BI semantic models; guided discovery, model investigation, and expert rule authoring.

    1k GitHub stars~4.4k tokensUpdated 2 days ago
    Auto-check passed

  • Pbip

    data-goblin/power-bi-agentic-development

    Guidance for Power BI Project (PBIP) structure, thick and thin reports, project renames, forks, and validation.

    1k GitHub stars~3.8k tokensUpdated 2 days ago
    Auto-check passed

  • Review Report

    data-goblin/power-bi-agentic-development

    Actionable feedback on the quality, usage, and effectiveness of Power BI reports.

    1k GitHub stars~4.2k tokensUpdated 2 days ago
    Auto-check passed

  • Semantic Model

    data-goblin/power-bi-agentic-development

    This skill should be used whenever the user mentions a "semantic model", "data model", or "dataset", or asks to "build", "model", "design", "optimize", "review", or "audit" one, or to "add a…

    1k GitHub stars~2.8k tokensUpdated 2 days ago
    Auto-check passed

Works with

Categories

Questions about Executing Spark

What does Executing Spark do?

Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Executing Spark is an agent skill from data-goblin/power-bi-agentic-development. Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access.

When should I use Executing Spark?

Executing Spark fits situations like: asks to run PySpark in Fabric; create a Livy session; execute Python on Fabric compute; run Spark without a notebook.

How do I install Executing Spark in Claude Code?

Run `npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -a claude-code`. Or copy the skill folder (plugins/etl/skills/executing-spark in data-goblin/power-bi-agentic-development) into .claude/skills/executing-spark in your project. Claude Code loads it when a task matches its description.

How do I install Executing Spark in Codex?

Run `npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -a codex`. Or copy the skill folder (plugins/etl/skills/executing-spark in data-goblin/power-bi-agentic-development) into .agents/skills/executing-spark in your project. Codex loads it when a task matches its description.

Can I use Executing Spark in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/executing-spark, .gemini/skills/executing-spark, .github/skills/executing-spark and .opencode/skills/executing-spark in your project.

What does Executing Spark need to run?

Going by SKILL.md and its folder, Executing Spark needs the command-line tools its instructions call (az). Our summary lists: Python 3.

Does Executing Spark access the network?

SKILL.md names 1 domain. In commands or code: api.fabric.microsoft.com; the agent is likely to contact it when it follows the instructions. This is read from the text; nothing was executed.

Is Executing Spark safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Executing Spark use?

Executing Spark is published under the GPL-3.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Executing Spark use?

About 1.7k tokens (SKILL.md is roughly 6.6k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 2.2k tokens, read only when the agent opens those files.

What are the alternatives to Executing Spark?

Skills that share tags, products or a category with Executing Spark: Apache Spark Optimization (wshobson/agents, 40k stars), Spark Engineer (FerroxLabs/wayland, 608 stars), Crawl4AI Web Scraping (smallnest/goclaw, 598 stars) and Migrate Glue Devendpoint To Interactive Sessions (aws-samples/aws-glue-samples, 1.5k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Executing Spark?

data-goblin (a GitHub user) maintains it in data-goblin/power-bi-agentic-development, which has 1,026 GitHub stars. The repository holds 33 skills in this directory. The repository was last updated on October 5, 2026.

Source: data-goblin/power-bi-agentic-development on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.