Migrate Glue Devendpoint To Interactive Sessions
aws-samples/aws-glue-samples
Migrate a legacy AWS Glue development endpoint to a Glue interactive session, following the official AWS migration checklist.
Create a new PySpark data source implementation. An agent skill from allisonwang-db/pyspark-data-sources.
Install Create Data Source
$ npx skills add allisonwang-db/pyspark-data-sources --skill create-data-source -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install allisonwang-db/pyspark-data-sources create-data-source --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/allisonwang-db/pyspark-data-sources.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.cursor/skills/create-data-source .claude/skills/create-data-source && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "create-data-source" agent skill from https://github.com/allisonwang-db/pyspark-data-sources/tree/master/.cursor/skills/create-data-source into .claude/skills/create-data-source/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "create-data-source", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/allisonwang-db/pyspark-data-sources/tree/master/.cursor/skills/create-data-sourceType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add allisonwang-db/pyspark-data-sources --skill create-data-source -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install allisonwang-db/pyspark-data-sources create-data-source --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/allisonwang-db/pyspark-data-sources.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.cursor/skills/create-data-source .agents/skills/create-data-source && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "create-data-source" agent skill from https://github.com/allisonwang-db/pyspark-data-sources/tree/master/.cursor/skills/create-data-source into .agents/skills/create-data-source/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "create-data-source", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add allisonwang-db/pyspark-data-sources --skill create-data-source -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install allisonwang-db/pyspark-data-sources create-data-source --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/allisonwang-db/pyspark-data-sources.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.cursor/skills/create-data-source .cursor/skills/create-data-source && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "create-data-source" agent skill from https://github.com/allisonwang-db/pyspark-data-sources/tree/master/.cursor/skills/create-data-source into .cursor/skills/create-data-source/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "create-data-source", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/allisonwang-db/pyspark-data-sources.git --path .cursor/skills/create-data-source--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add allisonwang-db/pyspark-data-sources --skill create-data-source -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install allisonwang-db/pyspark-data-sources create-data-source --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/allisonwang-db/pyspark-data-sources.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.cursor/skills/create-data-source .gemini/skills/create-data-source && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "create-data-source" agent skill from https://github.com/allisonwang-db/pyspark-data-sources/tree/master/.cursor/skills/create-data-source into .gemini/skills/create-data-source/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "create-data-source", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install allisonwang-db/pyspark-data-sources create-data-sourceInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add allisonwang-db/pyspark-data-sources --skill create-data-source -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/allisonwang-db/pyspark-data-sources.git skills-src && mkdir -p .github/skills && cp -r skills-src/.cursor/skills/create-data-source .github/skills/create-data-source && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "create-data-source" agent skill from https://github.com/allisonwang-db/pyspark-data-sources/tree/master/.cursor/skills/create-data-source into .github/skills/create-data-source/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "create-data-source", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add allisonwang-db/pyspark-data-sources --skill create-data-source -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install allisonwang-db/pyspark-data-sources create-data-source --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/allisonwang-db/pyspark-data-sources.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.cursor/skills/create-data-source .opencode/skills/create-data-source && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "create-data-source" agent skill from https://github.com/allisonwang-db/pyspark-data-sources/tree/master/.cursor/skills/create-data-source into .opencode/skills/create-data-source/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "create-data-source", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
create-data-source- GitHub stars
- 106
- Token cost
- ~635 tokens
- SKILL.md length
- 266 words
- Files
- 3
- Skills in repo
- 1
- Repo updated
- First seen
- Licence
- Apache-2.0
At a glance
Create a new PySpark data source implementation. An agent skill from allisonwang-db/pyspark-data-sources.
- Works in 5 steps: Create Implementation File → Register Data Source → Add Dependencies → …
- Adding a new connector
- SKILL.md covers Overview, Workflow, Implementation Details and Checklist, plus 1 more section
- Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md
What it does
Create Data Source is an agent skill from allisonwang-db/pyspark-data-sources. Create a new PySpark data source implementation. Use when adding a new connector, data source, or integration to the project.
Its SKILL.md is about 640 tokens, which your agent loads only when the skill is triggered. The skill folder holds 2 other files (for example `checklist.md` and `templates.md`).
It sits in Data & Analytics. It works with Apache Spark. The repository describes itself as: Custom PySpark Connectors. The licence is Apache-2.0.
When your agent uses it
- Adding a new connector
- Integration to the project
Example prompts
- “/create-data-source”
Requirements
- Python 3
Workflow steps
5 steps, taken from the step headings in SKILL.md.
What it can do on your machine
Read from SKILL.md and the folder at commit bf97b45. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
No scripts in the folder and no shell commands in SKILL.md (its code samples are python).
From the folder's file list and the shell code blocks in SKILL.md.
Network
Links to these hosts (documentation or services it may open):
spark.apache.org
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Create Data Source loads about 635 tokens when it runs. Until then it costs about 36 tokens; SKILL.md has 266 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from allisonwang-db/pyspark-data-sources at commit bf97b45, republished under its Apache-2.0 licence (© allisonwang-db). 266 words, ~635 tokens.
Download SKILL.mdSave it as
.claude/skills/create-data-source/SKILL.md (or your agent's skills folder). This skill also uses 2 other files; get the full folder from GitHub.- name
- create-data-source
- description
- Create a new PySpark data source implementation. Use when adding a new connector, data source, or integration to the project.
Create Data Source
Overview
This skill guides you through the process of adding a new PySpark data source to the repository. A data source typically consists of:
- DataSource: The main entry point, defining capabilities and schema.
- Reader: Logic for reading data (batch/stream).
- Writer: Logic for writing data (batch/stream).
- Tests: Unit and integration tests.
- Documentation: Usage guide and API reference.
Workflow
- Define Requirements: Determine if the source supports reading, writing, or both. Is it batch or streaming?
- Implementation: Create the implementation file in
pyspark_datasources/. - Registration: Register the new source in
pyspark_datasources/__init__.py. - Dependencies: Add any required libraries to
pyproject.toml. - Testing: Create a test file in
tests/. - Documentation: Add documentation in
docs/datasources/and updatemkdocs.ymlandREADME.md.
Implementation Details
1. Create Implementation File
Create a new file pyspark_datasources/<name>.py. Use the templates in templates.md.
- Implement
DataSourceclass. - Implement
DataSourceReader(if reading). - Implement
DataSourceWriter(if writing). - Define the schema in the
DataSourceclass.
2. Register Data Source
Add the new class to pyspark_datasources/__init__.py:
from .<name> import <Name>DataSource3. Add Dependencies
If the data source requires external libraries:
- Add them to
[project.optional-dependencies]inpyproject.toml. - Update the
allgroup to include the new dependencies.
4. Add Tests
Create tests/test_<name>.py.
- Use
unittest.mockto mock external services/libraries. - Test registration, reading, and writing logic.
- See
templates.mdfor test structure.
5. Add Documentation
- Create
docs/datasources/<name>.md. - Add the new page to
navinmkdocs.yml. - Add installation and usage examples to
README.mdanddocs/data-sources-guide.md.
Checklist
Use the checklist in checklist.md to track your progress.
Resources
- Python Data Source API Documentation
- Existing implementations in
pyspark_datasources/(e.g.,github.py,salesforce.py).
© allisonwang-db, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
SKILL.md and 2 other files in .cursor/skills/create-data-source of allisonwang-db/pyspark-data-sources.
- SKILL.md
- checklist.md
- templates.md
Open the folder on GitHubat commit bf97b45
Compare with similar skills
Create Data Source next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Create Data Source this skillallisonwang-db/pyspark-data-sources | 106 | — | ~635 | Automated safety check: Pass | Apache-2.0 | |
| Migrate Glue Devendpoint To Interactive Sessionsaws-samples/aws-glue-samples | 1.5k | — | ~3.6k | Automated safety check: Pass | MIT-0 | |
| Datafusion Pythonapache/datafusion-python | 606 | — | ~7.8k | Automated safety check: Pass | Apache-2.0 | |
| Apache Spark EngineerJeffallan/claude-skills | 12k | 1 repos | ~1.7k | Automated safety check: Pass | MIT | |
| Apache Spark Optimizationwshobson/agents | 40k | 8 repos | ~789 | Automated safety check: Pass | MIT | |
| Make Pythonicapache/datafusion-python | 606 | — | ~5.8k | Automated safety check: Pass | Apache-2.0 |
Similar skills
- Official
Datafusion Python
apache/datafusion-python
A skill your agent uses when the user is writing datafusion-python (Apache DataFusion Python bindings) DataFrame or SQL code.
Apache Spark Engineer
Jeffallan/claude-skills
Guides writing and tuning Apache Spark jobs: DataFrame and RDD code, Spark SQL, partitioning, caching, shuffle tuning and structured streaming.
Apache Spark Optimization
wshobson/agents
Speed up slow Apache Spark jobs by tuning partitions, shuffles, data skew, caching and executor memory, with do and don't rules for PySpark code.
Make Pythonic
apache/datafusion-python
Audit and improve datafusion-python functions to accept native Python types (int, float, str, bool) instead of requiring explicit lit() or col() wrapping.
Data Engineer
davila7/claude-code-templates
Build scalable data pipelines, modern data warehouses, and real-time streaming architectures.
Related
Works with
Categories
Questions about Create Data Source
What does Create Data Source do?
Create a new PySpark data source implementation. An agent skill from allisonwang-db/pyspark-data-sources. Create Data Source is an agent skill from allisonwang-db/pyspark-data-sources. Create a new PySpark data source implementation.
When should I use Create Data Source?
Create Data Source fits situations like: adding a new connector; integration to the project.
How do I install Create Data Source in Claude Code?
Run `npx skills add allisonwang-db/pyspark-data-sources --skill create-data-source -a claude-code`. Or copy the skill folder (.cursor/skills/create-data-source in allisonwang-db/pyspark-data-sources) into .claude/skills/create-data-source in your project. Claude Code loads it when a task matches its description.
How do I install Create Data Source in Codex?
Run `npx skills add allisonwang-db/pyspark-data-sources --skill create-data-source -a codex`. Or copy the skill folder (.cursor/skills/create-data-source in allisonwang-db/pyspark-data-sources) into .agents/skills/create-data-source in your project. Codex loads it when a task matches its description.
Can I use Create Data Source in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add allisonwang-db/pyspark-data-sources --skill create-data-source -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/create-data-source, .gemini/skills/create-data-source, .github/skills/create-data-source and .opencode/skills/create-data-source in your project.
What does Create Data Source need to run?
SKILL.md names no scripts, command-line tools or credentials: Create Data Source is instructions for the agent only. Our summary lists: Python 3.
Does Create Data Source access the network?
SKILL.md names 1 domain. As links in the text: spark.apache.org. This is read from the text; nothing was executed.
Is Create Data Source safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does Create Data Source use?
Create Data Source is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Create Data Source use?
About 635 tokens (SKILL.md is roughly 2.5k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to Create Data Source?
Skills that share tags, products or a category with Create Data Source: Migrate Glue Devendpoint To Interactive Sessions (aws-samples/aws-glue-samples, 1.5k stars), Datafusion Python (apache/datafusion-python, 606 stars), Apache Spark Engineer (Jeffallan/claude-skills, 12k stars) and Apache Spark Optimization (wshobson/agents, 40k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Create Data Source?
allisonwang-db (a GitHub user) maintains it in allisonwang-db/pyspark-data-sources, which has 106 GitHub stars. The repository was last updated on August 21, 2026.
Source: allisonwang-db/pyspark-data-sources on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.