Apache Spark Engineer

What it does

When your agent uses it

• Writing Spark jobs and DataFrame transformations
• Tuning shuffle partitions, executor memory or other cluster settings
• Fixing data skew or slow joins in a Spark pipeline
• Building a structured streaming job with watermarks and stateful logic

Example prompts

• “Rewrite this PySpark job to use a broadcast join for the country lookup table.”
• “My Spark stage spills to disk during the shuffle. Help me tune partitions and memory.”
• “Write a structured streaming job that reads new files from a landing folder and counts events per five-minute window with a watermark.”
• “Change the nightly Parquet processing script to write partitioned output.”

Requirements

• Apache Spark or PySpark

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

1. 1Analyze requirements - Understand data volume, transformations, latency requirements, cluster resources
2. 2Design pipeline - Choose DataFrame vs RDD, plan partitioning strategy, identify broadcast opportunities
3. 3Implement - Write Spark code with optimized transformations, appropriate caching, proper error handling
4. 4Optimize - Analyze Spark UI, tune shuffle partitions, eliminate skew, optimize joins and aggregations
5. 5Validate - Check Spark UI for shuffle spill before proceeding; verify partition count with df.rdd.getNumPartitions(); if spill or skew…

What it can do on your machine

Read from SKILL.md and the folder at commit 1be15d8. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  No scripts in the folder and no shell commands in SKILL.md (its code samples are python).

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Links to these hosts (documentation or services it may open):

  • github.com
  • synergetic.solutions
  • jeffallan.github.io

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Apache Spark Engineer loads about 1.7k tokens when it runs, and up to ~22k if it reads all its reference files. Until then it costs about 109 tokens; SKILL.md has 398 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~109

When it runs · the whole SKILL.md, loaded when a task matches

~1.7k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~22k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from Jeffallan/claude-skills at commit 1be15d8, republished under its MIT licence (© Jeffallan). 398 words, ~1,679 tokens.

name

spark-engineer

description

Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.

license

MIT

metadata.author

https://github.com/Jeffallan

metadata.company

https://synergetic.solutions

metadata.version

1.1.0

metadata.domain

data-ml

metadata.triggers

Apache Spark, PySpark, Spark SQL, distributed computing, big data, DataFrame API, RDD, Spark Streaming, structured streaming, data partitioning, Spark…

metadata.role

expert

metadata.scope

implementation

metadata.output-format

code

metadata.related-skills

python-pro, sql-pro, devops-engineer

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import StructType, StructField, StringType, LongType, DoubleType

spark = SparkSession.builder \
    .appName("example-pipeline") \
    .config("spark.sql.shuffle.partitions", "400") \
    .config("spark.sql.adaptive.enabled", "true") \
    .getOrCreate()

# Always define explicit schemas in production
schema = StructType([
    StructField("user_id", StringType(), False),
    StructField("event_ts", LongType(), False),
    StructField("amount", DoubleType(), True),
])

df = spark.read.schema(schema).parquet("s3://bucket/events/")

result = df \
    .filter(F.col("amount").isNotNull()) \
    .groupBy("user_id") \
    .agg(F.sum("amount").alias("total_amount"), F.count("*").alias("event_count"))

# Verify partition count before writing
print(f"Partition count: {result.rdd.getNumPartitions()}")

result.write.mode("overwrite").parquet("s3://bucket/output/")

from pyspark.sql.functions import broadcast

# Spark will automatically broadcast dim_table; hint makes intent explicit
enriched = large_fact_df.join(broadcast(dim_df), on="product_id", how="left")

import pyspark.sql.functions as F

SALT_BUCKETS = 50

# Add salt to the skewed key on both sides
skewed_df = skewed_df.withColumn("salt", (F.rand() * SALT_BUCKETS).cast("int")) \
    .withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))

other_df = other_df.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(SALT_BUCKETS)]))) \
    .withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))

result = skewed_df.join(other_df, on="salted_key", how="inner") \
    .drop("salt", "salted_key")

# Cache ONLY when the DataFrame is reused multiple times
df_cleaned = df.filter(...).withColumn(...).cache()
df_cleaned.count()  # Materialize immediately; check Spark UI for spill

report_a = df_cleaned.groupBy("region").agg(...)
report_b = df_cleaned.groupBy("product").agg(...)

df_cleaned.unpersist()  # Release when done

© Jeffallan, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 5 other files (references) in skills/spark-engineer of Jeffallan/claude-skills.

• SKILL.md
• references/partitioning-caching.md
• references/performance-tuning.md
• references/rdd-operations.md
• references/spark-sql-dataframes.md
• references/streaming-patterns.md

Open the folder on GitHubat commit 1be15d8

Used in 1 other repository

We found 1 copy of this SKILL.md (exact, near-identical or edited) in other folders, from 1 other GitHub owner. This page covers the copy in Jeffallan/claude-skills, which our catalogue first saw on October 7, 2026.

• Kilo-Org/kilo-marketplaceskills/spark-engineer