New: 97,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Add Source Datastream To Spanner

by GoogleCloudPlatform in GoogleCloudPlatform/DataflowTemplates

Guide for implementing a database source connector in the v2/datastream-to-spanner forward migration Dataflow template.

Apache-2.0Auto-check: notesTesting & QA

Install Add Source Datastream To Spanner

skills CLI
$ npx skills add GoogleCloudPlatform/DataflowTemplates --skill add-source-datastream-to-spanner -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install GoogleCloudPlatform/DataflowTemplates add-source-datastream-to-spanner --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/GoogleCloudPlatform/DataflowTemplates.git skills-src && mkdir -p .claude/skills && cp -r skills-src/v2/datastream-to-spanner/.agents/skills/add-source-datastream-to-spanner .claude/skills/add-source-datastream-to-spanner && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
add-source-datastream-to-spanner
GitHub stars
1.3k
Token cost
~1.7k tokens
SKILL.md length
778 words
Files
1
Skills in repo
12
Repo updated
First seen
Licence
Apache-2.0

At a glance

Guide for implementing a database source connector in the v2/datastream-to-spanner forward migration Dataflow template.

  • Works in 6 steps: Mandatory Prerequisite Gate → Implement DsToSpSourceConnector → Implement Source Processing Classes → …
  • Tasks that involve Unit testing
  • SKILL.md covers Overview, Prerequisites, Architectural Boundaries &… and Datatype Mapping Matrix…, plus 1 more section
  • Calls mvn

What it does

Add Source Datastream To Spanner is an agent skill from GoogleCloudPlatform/DataflowTemplates. Guide for implementing a database source connector in the v2/datastream-to-spanner forward migration Dataflow template. Details scope boundaries, prerequisites, connector implementations, registry registrations, unit testing, and smoke testing guidelines.

Its SKILL.md is about 1.7k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in Testing & QA, covering Unit testing. It works with Google Cloud and Google BigQuery. The repository describes itself as: Cloud Dataflow Google-provided templates for solving in-Cloud data tasks. The licence is Apache-2.0.

When your agent uses it

  • Tasks that involve Unit testing

Example prompts

  • “/add-source-datastream-to-spanner”

Workflow steps

6 steps, taken from the step headings in SKILL.md.

  1. Mandatory Prerequisite Gate
  2. Implement DsToSpSourceConnector
  3. Implement Source Processing Classes
  4. Register Source in Registries & Config Files
  5. Unit Testing & Verification
  6. Mandatory Live Smoke Testing & Verification

What it can do on your machine

Read from SKILL.md and the folder at commit c95daba. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • mvn

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Links to these hosts (documentation or services it may open):

    • docs.cloud.google.com

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Add Source Datastream To Spanner loads about 1.7k tokens when it runs. Until then it costs about 72 tokens; SKILL.md has 778 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~72
When it runs · the whole SKILL.md, loaded when a task matches
~1.7k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check: notes

The automated check noted patterns worth knowing about, such as sudo or a known installer.

  • NoteMentions a .env fileSKILL.md:90
    not explicitly specified by the user or `.env.testing`, **you MUST ALWAYS default to `--worker-machine-type=n2-standard-

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from GoogleCloudPlatform/DataflowTemplates at commit c95daba, republished under its Apache-2.0 licence (© GoogleCloudPlatform). 778 words, ~1,733 tokens.

Download SKILL.mdSave it as .claude/skills/add-source-datastream-to-spanner/SKILL.md (or your agent's skills folder).
name
add-source-datastream-to-spanner
description
Guide for implementing a database source connector in the v2/datastream-to-spanner forward migration Dataflow template. Details scope boundaries, prerequisites, connector implementations, registry registrations, unit testing, and smoke testing guidelines.

Skill: Implement Source in Datastream to Spanner Template

Overview

This skill provides a step-by-step procedure for adding support for a new database source connector to the v2/datastream-to-spanner template. It details prerequisites, scope boundaries, type mapping requirements, connector implementation, registry registration, unit testing, and smoke testing procedures.

Prerequisites

[!CRITICAL] MANDATORY PREREQUISITE GATE: Before running code searches, inspecting files, or executing unit tests, you MUST verify if the user provided the inputs below. If any required input is missing, STOP IMMEDIATELY and ask the user for clarification before proceeding further.

  1. Datastream Source Compatibility Check: Verify that Google Cloud Datastream natively supports the requested source database type by referencing Datastream Supported Sources.

    • [!CRITICAL]

    • If the requested source database is NOT supported by Datastream, STOP IMMEDIATELY and inform the user that Datastream does not support CDC streaming for this database source. Do NOT proceed with inspection, implementation, or testing.

  2. Datatype Mapping File: Ask the user to provide or point to the Datatype Mapping Matrix for the new source database, defining the mappings to Spanner GoogleSQL and PostgreSQL dialects.
  3. Test Setup Details: Ask the user for the test environment details required for live smoke testing, including:
    • Source Database Instance details (host/instance name, database name, and credentials/connection method).
    • Target Spanner Instance & Database details.
    • Datastream stream / connection profile setup or GCP project information.

Architectural Boundaries & Code Scope

All implementation for a new source connector MUST be strictly confined to:

  1. Source Connector Package: v2/datastream-to-spanner/src/main/java/com/google/cloud/teleport/v2/templates/source/<source_type>/
    • Implementation of IDsToSpSourceConnector (<Source>DsToSpSourceConnector.java)
    • Source-specific change event processing classes in package <source_type> (implemented similar to other existing source connectors in the codebase).
  2. Connector Registry: DatastreamToSpannerSourceConnectorRegistry.java
    • Dynamic connector registration via register(new <Source>DsToSpSourceConnector()).
  3. Shared Core Registries & Configs: v2/spanner-common
    • Registries and configuration files in spanner-common (updated similar to existing sources).

Datatype Mapping Matrix Requirements

Consult the Datatype Mapping Matrix provided in the prerequisites to verify correct datatype conversion to Spanner GoogleSQL and PostgreSQL dialects. Ensure proper alignment for character, numeric, temporal, binary, boolean, JSON and any other datatypes as specified in the mapping file.

Step-by-Step Implementation Workflow

Step 0: Mandatory Prerequisite Gate
  1. Verify Datastream Support: Check if the requested source database type is supported by Datastream per Datastream Sources. If unsupported, STOP IMMEDIATELY and notify the user.
  2. Verify Inputs: Inspect the request for the required inputs. If the Datatype Mapping Matrix and Test Setup Details are not provided in the user request, DO NOT run any inspection or execution tools. Stop and ask the user for the missing details first.
Step 1: Implement <Source>DsToSpSourceConnector

Implement IDsToSpSourceConnector in com.google.cloud.teleport.v2.templates.source.<source_type>:

  1. Define CDC metadata key constants required by the source.
  2. Implement any other methods in this class to work for GoogleSQL and PostgreSQL Spanner dialects similar to other supported sources.
Show full SKILL.md (331 more words)Show less
Step 2: Implement Source Processing Classes

Implement remaining source-specific change event context and sequence classes in package com.google.cloud.teleport.v2.templates.source.<source_type> similar to other existing source connectors. Skip the types that are not supported by Datastream. These are mentioned in the type mapping skill.

Step 3: Register Source in Registries & Config Files
  1. Register connector in DatastreamToSpannerSourceConnectorRegistry.java:
    java
    register(new <Source>DsToSpSourceConnector());
  2. Update corresponding registries and converter files in v2/spanner-common similar to existing sources.
Step 4: Unit Testing & Verification

Execute Maven unit tests for the template:

bash
mvn test -pl v2/datastream-to-spanner \
  -Dtest=<Source>DsToSpSourceConnectorTest,DatastreamToSpannerSourceConnectorRegistryTest

All unit tests must pass with BUILD SUCCESS.

Step 5: Mandatory Live Smoke Testing & Verification

[!CRITICAL] MANDATORY EXECUTION REQUIREMENT: Immediately after unit tests pass, you MUST AUTOMATICALLY PROCEED to execute live end-to-end smoke testing using the test environment details provided in the prerequisites. Do NOT stop, pause, or declare completion after unit testing without running the live smoke tests.

[!IMPORTANT] WORKER MACHINE TYPE DIRECTIVE & FALLBACK RULE: When submitting the Dataflow job for live smoke testing, you MUST explicitly specify a worker machine type of the correct size with at least 4 vCPUs. If --worker-machine-type (or WORKER_MACHINE_TYPE) is not explicitly specified by the user or .env.testing, you MUST ALWAYS default to --worker-machine-type=n2-standard-4. Omitting this parameter or submitting null/blank will cause Dataflow job launch validation to fail with a machine specification policy violation.

  1. Environment Setup:
    • Connect to the live source database instance and target Spanner instance configured in the test setup.
  2. Perform CRUD Operations:
    • INSERT: Execute INSERT statements covering supported datatypes in the source database.
    • UPDATE: Update existing rows in the source database.
    • DELETE: Delete test rows from the source database.
  3. Verify Replication Flow:
    • Verify that CDC events flow through Datastream to Cloud Storage and Dataflow.
    • Query Cloud Spanner tables to confirm that INSERT, UPDATE, and DELETE operations are accurately reflected in Spanner.
  4. Retry Loop on Failure:
    • If any CRUD operation fails to replicate or produces data discrepancies in Spanner, inspect error logs, modify the connector and converter code, rebuild, and re-test until all operations pass cleanly.

© GoogleCloudPlatform, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in v2/datastream-to-spanner/.agents/skills/add-source-datastream-to-spanner of GoogleCloudPlatform/DataflowTemplates.

Open the folder on GitHubat commit c95daba

Compare with similar skills

Add Source Datastream To Spanner next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Add Source Datastream To Spanner compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Add Source Datastream To Spanner this skillGoogleCloudPlatform/DataflowTemplates1.3k—~1.7kAutomated safety check: NotesApache-2.0
Adding Dbt Unit TestKilo-Org/kilo-marketplace189—~4.2kAutomated safety check: PassApache-2.0
Payload Snapshotopenshift-eng/ai-helpers120—~6.4kAutomated safety check: PassApache-2.0
Io ConnectorsKilo-Org/kilo-marketplace189—~1.3kAutomated safety check: PassApache-2.0
Jobs And Testsopenshift-eng/ai-helpers120—~2.2kAutomated safety check: PassApache-2.0
Retail Product Search Agentgoogle/adk-recipes10k—~3kAutomated safety check: PassApache-2.0

Similar skills

  • Adding Dbt Unit Test

    Kilo-Org/kilo-marketplace

    Creates unit test YAML definitions that mock upstream model inputs and validate expected outputs.

    189 GitHub stars~4.2k tokensUpdated 9 days ago
    Testing & QAAuto-check passed

  • Payload Snapshot

    openshift-eng/ai-helpers

    Snapshot OpenShift payload data (release controller, PR diffs, comments, CI jobs, JUnit results, regression tracking) to a local directory for offline analysis

    120 GitHub stars~6.4k tokensUpdated today
    Testing & QAAuto-check passed

  • Io Connectors

    Kilo-Org/kilo-marketplace

    Guides development and usage of I/O connectors in Apache Beam.

    189 GitHub stars~1.3k tokensUpdated 9 days ago
    Testing & QAAuto-check passed

  • Jobs And Tests

    openshift-eng/ai-helpers

    A skill your agent uses when querying OpenShift CI prow job runs and junit test results in BigQuery (cianalysisus dataset) with deduplication, variant joins, and infrastructure failure filtering

    120 GitHub stars~2.2k tokensUpdated today
    DatabasesAuto-check passed

  • Retail Product Search Agent

    google/adk-recipes

    Official

    Builds a retail product search agent on Google Cloud, from catalog ingestion into BigQuery and Vector Search to ADK scaffolding, evaluation and Cloud Run deployment.

    10k GitHub stars~3k tokensUpdated yesterday
    AI & LLM EngineeringAuto-check passed

  • Cxas Configurable Dashboards

    GoogleCloudPlatform/cxas-scrapi

    Author, validate, and manage Contact Center AI (CCAI) Insights Configurable Dashboards.

    106 GitHub stars~1.8k tokensUpdated today
    DatabasesAuto-check passed

More from GoogleCloudPlatform/DataflowTemplates

All 12 skills in this repo

  • Smt E2E Dataflow Debugging

    GoogleCloudPlatform/DataflowTemplates

    Debugs logical errors and data discrepancies in Dataflow templates by launching jobs via Terraform and comparing source (e.g.

    1.3k GitHub stars~1.8k tokensUpdated today
    Auto-check passed

  • Smt Functional Testing

    GoogleCloudPlatform/DataflowTemplates

    Functionally tests local Dataflow pipeline changes against the main branch using ephemeral GCP resources and gated approvals.

    1.3k GitHub stars~2.8k tokensUpdated today
    Auto-check: notes

  • Add Integ Tests Datastream To Spanner

    GoogleCloudPlatform/DataflowTemplates

    Specific runner skill that delegates to the Template-Agnostic Meta-Test Orchestrator for the datastream-to-spanner (CDC) template.

    1.3k GitHub stars~572 tokensUpdated today
    Auto-check passed

  • Add Integ Tests Gcs Spanner Dv

    GoogleCloudPlatform/DataflowTemplates

    Specific runner skill that creates integration tests for the gcs-spanner-dv (Data Validation) template.

    1.3k GitHub stars~517 tokensUpdated today
    Auto-check passed

  • Add Integ Tests Sourcedb To Spanner

    GoogleCloudPlatform/DataflowTemplates

    Specific runner skill that delegates to the Template-Agnostic Meta-Test Orchestrator for the sourcedb-to-spanner (Bulk) template.

    1.3k GitHub stars~475 tokensUpdated today
    Auto-check passed

  • Add Integ Tests Spanner To Sourcedb

    GoogleCloudPlatform/DataflowTemplates

    Specific runner skill that delegates to the Template-Agnostic Meta-Test Orchestrator for the spanner-to-sourcedb (Reverse Migration) template.

    1.3k GitHub stars~479 tokensUpdated today
    Auto-check passed

Works with

Categories

Questions about Add Source Datastream To Spanner

What does Add Source Datastream To Spanner do?

Guide for implementing a database source connector in the v2/datastream-to-spanner forward migration Dataflow template. Add Source Datastream To Spanner is an agent skill from GoogleCloudPlatform/DataflowTemplates. Guide for implementing a database source connector in the v2/datastream-to-spanner forward migration Dataflow template.

When should I use Add Source Datastream To Spanner?

Add Source Datastream To Spanner fits situations like: tasks that involve Unit testing.

How do I install Add Source Datastream To Spanner in Claude Code?

Run `npx skills add GoogleCloudPlatform/DataflowTemplates --skill add-source-datastream-to-spanner -a claude-code`. Or copy the skill folder (v2/datastream-to-spanner/.agents/skills/add-source-datastream-to-spanner in GoogleCloudPlatform/DataflowTemplates) into .claude/skills/add-source-datastream-to-spanner in your project. Claude Code loads it when a task matches its description.

How do I install Add Source Datastream To Spanner in Codex?

Run `npx skills add GoogleCloudPlatform/DataflowTemplates --skill add-source-datastream-to-spanner -a codex`. Or copy the skill folder (v2/datastream-to-spanner/.agents/skills/add-source-datastream-to-spanner in GoogleCloudPlatform/DataflowTemplates) into .agents/skills/add-source-datastream-to-spanner in your project. Codex loads it when a task matches its description.

Can I use Add Source Datastream To Spanner in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add GoogleCloudPlatform/DataflowTemplates --skill add-source-datastream-to-spanner -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/add-source-datastream-to-spanner, .gemini/skills/add-source-datastream-to-spanner, .github/skills/add-source-datastream-to-spanner and .opencode/skills/add-source-datastream-to-spanner in your project.

What does Add Source Datastream To Spanner need to run?

Going by SKILL.md and its folder, Add Source Datastream To Spanner needs the command-line tools its instructions call (mvn).

Does Add Source Datastream To Spanner access the network?

SKILL.md names 1 domain. As links in the text: docs.cloud.google.com. This is read from the text; nothing was executed.

Is Add Source Datastream To Spanner safe to install?

Our automated static check of SKILL.md found notes only (mentions a .env file), nothing it rates as a warning. It is not a guarantee. Review the folder before installing.

What licence does Add Source Datastream To Spanner use?

Add Source Datastream To Spanner is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Add Source Datastream To Spanner use?

About 1.7k tokens (SKILL.md is roughly 6.9k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Add Source Datastream To Spanner?

Skills that share tags, products or a category with Add Source Datastream To Spanner: Adding Dbt Unit Test (Kilo-Org/kilo-marketplace, 189 stars), Payload Snapshot (openshift-eng/ai-helpers, 120 stars), Io Connectors (Kilo-Org/kilo-marketplace, 189 stars) and Jobs And Tests (openshift-eng/ai-helpers, 120 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Add Source Datastream To Spanner?

GoogleCloudPlatform (a GitHub organization) maintains it in GoogleCloudPlatform/DataflowTemplates, which has 1,315 GitHub stars. The repository holds 12 skills in this directory. The repository was last updated on October 7, 2026.

Source: GoogleCloudPlatform/DataflowTemplates on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.