New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Clickhouse Webhooks Events

by jeremylongshore in jeremylongshore/tons-of-skills-marketplace

Ingest data into ClickHouse from webhooks, Kafka, and streaming sources with batching, dedup, and exactly-once patterns.

MITAuto-check passedBackend & APIs

Install Clickhouse Webhooks Events

skills CLI
$ npx skills add jeremylongshore/tons-of-skills-marketplace --skill clickhouse-webhooks-events -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install jeremylongshore/tons-of-skills-marketplace clickhouse-webhooks-events --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/jeremylongshore/tons-of-skills-marketplace.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/.curated/clickhouse-webhooks-events .claude/skills/clickhouse-webhooks-events && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
clickhouse-webhooks-events
GitHub stars
2.8k
Token cost
~1.7k tokens
SKILL.md length
457 words
Files
3 (incl. references)
Skills in repo
3,342
Repo updated
First seen
Licence
MIT

At a glance

Ingest data into ClickHouse from webhooks, Kafka, and streaming sources with batching, dedup, and exactly-once patterns.

  • Works in 3 steps: Webhook Receiver with Batched Inserts → Choose a Server-Side or Managed Path → Make Ingestion Idempotent and Observable
  • Building data ingestion pipelines
  • SKILL.md covers Overview, Prerequisites, Instructions and Output, plus 4 more sections
  • Calls curl

What it does

Clickhouse Webhooks Events is an agent skill from jeremylongshore/tons-of-skills-marketplace. Ingest data into ClickHouse from webhooks, Kafka, and streaming sources with batching, dedup, and exactly-once patterns. Use when building data ingestion pipelines, consuming webhook payloads, or integrating Kafka topics into ClickHouse. Trigger with "clickhouse ingestion", "clickhouse webhook", "clickhouse Kafka", "stream data to clickhouse", "clickhouse data pipeline".

Its SKILL.md is about 1.7k tokens, which your agent loads only when the skill is triggered. The skill folder holds 3 other files, including reference files (for example `references/dedup-and-monitoring.md` and `references/ingestion-methods.md`). Compatibility notes: Designed for Claude Code

It sits in Backend & APIs, covering Webhooks, Data warehousing and Data pipelines and ETL. It works with ClickHouse and Apache Kafka. The repository describes itself as: Model-agnostic agent-skills platform with a harness-free canonical layer, verified adapters, and the ccpi package manager. Explore at tonsofskills.com. The licence is MIT.

When your agent uses it

  • Building data ingestion pipelines
  • Consuming webhook payloads
  • Integrating Kafka topics into ClickHouse
  • With clickhouse ingestion

Example prompts

  • “clickhouse ingestion”
  • “clickhouse webhook”
  • “clickhouse Kafka”
  • “/clickhouse-webhooks-events”

Requirements

  • Compatibility (from SKILL.md): Designed for Claude Code
  • Pre-approved tools (allowed-tools): Read, Write, Edit, Bash(curl:*)

Workflow steps

3 steps, taken from the step headings in SKILL.md.

  1. Webhook Receiver with Batched Inserts
  2. Choose a Server-Side or Managed Path
  3. Make Ingestion Idempotent and Observable

What it can do on your machine

Read from SKILL.md and the folder at commit cfae287. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves these tools, so the agent can use them without asking each time:

    • Read
    • Write
    • Edit
    • Bash(curl:*)

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • curl

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Links to these hosts (documentation or services it may open):

    • clickhouse.com

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

  • Compatibility

    Designed for Claude Code

    From compatibility in the SKILL.md frontmatter.

Context cost

Clickhouse Webhooks Events loads about 1.7k tokens when it runs, and up to ~2.9k if it reads all its reference files. Until then it costs about 100 tokens; SKILL.md has 457 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~100
When it runs · the whole SKILL.md, loaded when a task matches
~1.7k
With references · SKILL.md plus every file in references/, read only if the agent opens them
~2.9k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from jeremylongshore/tons-of-skills-marketplace at commit cfae287, republished under its MIT licence (© jeremylongshore). 457 words, ~1,658 tokens.

Download SKILL.mdSave it as .claude/skills/clickhouse-webhooks-events/SKILL.md (or your agent's skills folder). This skill also uses 2 other files; get the full folder from GitHub.
name
clickhouse-webhooks-events
description
Ingest data into ClickHouse from webhooks, Kafka, and streaming sources with batching, dedup, and exactly-once patterns. Use when building data ingestion pipelines, consuming webhook payloads, or integrating Kafka topics into ClickHouse. Trigger with "clickhouse ingestion", "clickhouse webhook", "clickhouse Kafka", "stream data to clickhouse", "clickhouse data pipeline".
allowed-tools
Read, Write, Edit, Bash(curl:*)
compatibility
Designed for Claude Code
version
1.7.0
license
MIT
author
Jeremy Longshore <jeremy@intentsolutions.io>
tags
saas, database, analytics, clickhouse, olap

ClickHouse Data Ingestion

Overview

Build data ingestion pipelines into ClickHouse from HTTP webhooks, Kafka, and streaming sources with proper batching, deduplication, and error handling.

The core rule: ClickHouse hates one-row-at-a-time inserts — buffer events and flush them in batches. This skill covers four ingestion paths (application-side webhook receiver, server-side Kafka engine, managed ClickPipes, and HTTP bulk loads) plus idempotent dedup and insert monitoring.

Prerequisites

  • A ClickHouse table with an appropriate engine already exists (a MergeTree variant, e.g. analytics.events) — see clickhouse-core-workflow-a.
  • The @clickhouse/client package is installed and connected via CLICKHOUSE_HOST.
  • For the Kafka paths, a reachable Kafka broker and topic.

Instructions

Step 1: Webhook Receiver with Batched Inserts

Buffer incoming events in memory, flush on a size threshold or a timer, and re-queue the batch on failure so no event is lost. This is the application-side core of the skill:

typescript
import express from 'express';
import { createClient } from '@clickhouse/client';

const client = createClient({ url: process.env.CLICKHOUSE_HOST! });
const app = express();
app.use(express.json());

// Buffer for batching — ClickHouse hates one-row-at-a-time inserts
const buffer: Record<string, unknown>[] = [];
const BATCH_SIZE = 5_000;
const FLUSH_INTERVAL_MS = 5_000;

async function flushBuffer() {
  if (buffer.length === 0) return;
  const batch = buffer.splice(0, buffer.length);

  try {
    await client.insert({
      table: 'analytics.events',
      values: batch,
      format: 'JSONEachRow',
    });
    console.log(`Flushed ${batch.length} events to ClickHouse`);
  } catch (err) {
    console.error('Insert failed, re-queuing:', (err as Error).message);
    buffer.unshift(...batch);  // Put back at front for retry
  }
}

// Flush periodically
setInterval(flushBuffer, FLUSH_INTERVAL_MS);

// Webhook endpoint
app.post('/ingest', async (req, res) => {
  const events = Array.isArray(req.body) ? req.body : [req.body];

  for (const event of events) {
    buffer.push({
      event_type: event.type ?? 'unknown',
      user_id: event.userId ?? 0,
      properties: JSON.stringify(event.properties ?? {}),
      created_at: new Date().toISOString().replace('T', ' ').slice(0, 19),
    });
  }

  if (buffer.length >= BATCH_SIZE) {
    await flushBuffer();
  }

  res.status(202).json({ queued: events.length, buffer_size: buffer.length });
});
Step 2: Choose a Server-Side or Managed Path

For high-volume streams, prefer a path that needs no application consumer:

  • Kafka table engine — ClickHouse consumes a topic directly and a materialized view pipes rows into your MergeTree table. No consumer to run.
  • ClickPipes — ClickHouse Cloud's managed, code-free ingestion for Kafka, Confluent, Amazon MSK, S3, and GCS.
  • HTTP interface — bulk-load CSV / NDJSON / Parquet from files, remote URLs, or S3 with plain curl, no client library.

Full DDL and configuration for all three: see Ingestion methods.

Step 3: Make Ingestion Idempotent and Observable

Webhook retries and Kafka reprocessing deliver duplicates. Use a ReplacingMergeTree keyed on a unique event_id so re-delivered events collapse to one row, and query system.query_log to watch insert throughput and errors. Full DDL, monitoring queries, and the batch-tuning matrix: Deduplication & monitoring.

Show full SKILL.md (188 more words)Show less

Output

Applying this skill produces:

  • A running webhook receiver (POST /ingest) that buffers events and batch-flushes to ClickHouse, returning 202 { queued, buffer_size }.
  • Optionally, a Kafka engine table + materialized view (or a ClickPipes pipe) that ingests a topic server-side with no application consumer.
  • A ReplacingMergeTree dedup table keyed on event_id for idempotent, retry-safe ingestion.
  • Monitoring queries over system.query_log reporting inserts/minute, rows, bytes, and insert exceptions in the last hour.

Error Handling

ErrorCauseSolution
Too many partsSingle-row insertsBatch inserts (10K+ rows)
Cannot parse inputWrong formatMatch format to data structure
TIMEOUT on large insertSlow networkEnable compression, split batch
Duplicate eventsWebhook retriesUse ReplacingMergeTree + event_id

Examples

Ingest a webhook batch via the receiver (Step 1):

bash
curl -X POST http://localhost:3000/ingest \
  -H 'Content-Type: application/json' \
  -d '[{"type":"signup","userId":42,"properties":{"plan":"pro"}}]'
# → 202 { "queued": 1, "buffer_size": 1 }

Bulk-load a Parquet file with no client (HTTP interface — see Ingestion methods):

bash
curl 'http://localhost:8123/?query=INSERT+INTO+analytics.events+FORMAT+Parquet' \
  --data-binary @events.parquet

Read deduplicated events (ReplacingMergeTree — see Deduplication & monitoring):

sql
SELECT * FROM analytics.events_dedup FINAL
WHERE created_at >= today() - 7;

Resources

Next Steps

For query and server performance after ingestion is flowing, see clickhouse-performance-tuning. For engine and schema choices on the target table, see clickhouse-core-workflow-a.

© jeremylongshore, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 2 other files (references) in skills/.curated/clickhouse-webhooks-events of jeremylongshore/tons-of-skills-marketplace.

  • SKILL.md
  • references/dedup-and-monitoring.md
  • references/ingestion-methods.md

Open the folder on GitHubat commit cfae287

Compare with similar skills

Clickhouse Webhooks Events next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Clickhouse Webhooks Events compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Clickhouse Webhooks Events this skilljeremylongshore/tons-of-skills-marketplace2.8k—~1.7kAutomated safety check: PassMIT
Querying TophogPostHog/posthog40k—~2.3kAutomated safety check: PassCustom licence
AWS Storageaws/agent-toolkit-for-aws2.8k—~5.8kAutomated safety check: PassApache-2.0
Opensourcefaqdigoal/blog8.6k—~966Automated safety check: PassGPL-2.0
Monitoring Ingestion PipelinePostHog/posthog40k—~9.1kAutomated safety check: PassCustom licence
Ar Io Gateway Operatorar-io/ar-io-node127—~7.7kAutomated safety check: NotesAGPL-3.0

Similar skills

  • Querying Tophog

    PostHog/posthog

    Official

    Query tophog — the ingestion pipeline's heavy-hitter store in ClickHouse — to identify hot or expensive actors (teamid, distinctid, sessionid, partition) during incident triage.

    40k GitHub stars~2.3k tokensUpdated yesterday
    Backend & APIsAuto-check passed

  • AWS Storage

    aws/agent-toolkit-for-aws

    Official

    Selects, investigates, and compares AWS object, file, and block storage services, and answers cost, performance, configuration, security, and troubleshooting questions about storage services.

    2.8k GitHub stars~5.8k tokensUpdated yesterday
    DatabasesAuto-check passed

  • Opensourcefaq

    digoal/blog

    解答与开源产品有关的深度技术问题，输出图文并茂的 Markdown 技术文章。触发条件：用户提出与开源项目（如 PostgreSQL、Redis、Kafka、Kubernetes、ClickHouse、Flink 等）相关的技术问题，并提供源码目录或 URL、deepwiki repo 名称。即使用户只说"帮我解答这个开源问题"或"分析一下这个项目的某个机制"，也应使用本…

    8.6k GitHub stars~966 tokensUpdated 2 days ago
    DatabasesAuto-check passed

  • Monitoring Ingestion Pipeline

    PostHog/posthog

    Official

    Guide for using the Grafana MCP to monitor and diagnose the Node.js ingestion pipeline workers in production.

    40k GitHub stars~9.1k tokensUpdated yesterday
    DevOps & CloudAuto-check passed

  • Ar Io Gateway Operator

    ar-io/ar-io-node

    Operate any AR.IO node deployment — architecture, daily ops, diagnostics, and recurring pitfalls that apply to every operator.

    127 GitHub stars~7.7k tokensUpdated today
    Backend & APIsAuto-check: notes

  • Create Environment

    godatadriven/whirl

    Create a new Whirl environment in the envs/ directory. An agent skill from godatadriven/whirl.

    205 GitHub stars~1.9k tokensUpdated 9 days ago
    Backend & APIsAuto-check passed

More from jeremylongshore/tons-of-skills-marketplace

All 3,342 skills in this repo

  • Performing Security Code Review

    jeremylongshore/tons-of-skills-marketplace

    Execute this skill enables AI assistant to conduct a security-focused code review using the security-agent plugin.

    2.8k GitHub starsUsed in 2 repos~1.3k tokens
    Auto-check: notes

  • Adapting Transfer Learning Models

    jeremylongshore/tons-of-skills-marketplace

    Build this skill automates the adaptation of pre-trained machine learning models using transfer learning techniques.

    2.8k GitHub stars~1.1k tokensUpdated today
    Auto-check passed

  • Agent Context Loader

    jeremylongshore/tons-of-skills-marketplace

    Execute proactive auto-loading: automatically detects and loads agents.md files.

    2.8k GitHub stars~1.1k tokensUpdated today
    Auto-check passed

  • Aggregating Performance Metrics

    jeremylongshore/tons-of-skills-marketplace

    Aggregate and centralize performance metrics from applications, systems, databases, caches, and services.

    2.8k GitHub stars~1.2k tokensUpdated today
    Auto-check passed

  • Analyzing Capacity Planning

    jeremylongshore/tons-of-skills-marketplace

    Execute this skill enables AI assistant to analyze capacity requirements and plan for future growth.

    2.8k GitHub stars~947 tokensUpdated today
    Auto-check passed

  • Analyzing Database Indexes

    jeremylongshore/tons-of-skills-marketplace

    Process use when you need to work with database indexing. An agent skill from jeremylongshore/tons-of-skills-marketplace.

    2.8k GitHub stars~2k tokensUpdated today
    Auto-check passed

Works with

Categories

Questions about Clickhouse Webhooks Events

What does Clickhouse Webhooks Events do?

Ingest data into ClickHouse from webhooks, Kafka, and streaming sources with batching, dedup, and exactly-once patterns. Clickhouse Webhooks Events is an agent skill from jeremylongshore/tons-of-skills-marketplace. Ingest data into ClickHouse from webhooks, Kafka, and streaming sources with batching, dedup, and exactly-once patterns.

When should I use Clickhouse Webhooks Events?

Clickhouse Webhooks Events fits situations like: building data ingestion pipelines; consuming webhook payloads; integrating Kafka topics into ClickHouse; with clickhouse ingestion.

How do I install Clickhouse Webhooks Events in Claude Code?

Run `npx skills add jeremylongshore/tons-of-skills-marketplace --skill clickhouse-webhooks-events -a claude-code`. Or copy the skill folder (skills/.curated/clickhouse-webhooks-events in jeremylongshore/tons-of-skills-marketplace) into .claude/skills/clickhouse-webhooks-events in your project. Claude Code loads it when a task matches its description.

How do I install Clickhouse Webhooks Events in Codex?

Run `npx skills add jeremylongshore/tons-of-skills-marketplace --skill clickhouse-webhooks-events -a codex`. Or copy the skill folder (skills/.curated/clickhouse-webhooks-events in jeremylongshore/tons-of-skills-marketplace) into .agents/skills/clickhouse-webhooks-events in your project. Codex loads it when a task matches its description.

Can I use Clickhouse Webhooks Events in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add jeremylongshore/tons-of-skills-marketplace --skill clickhouse-webhooks-events -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/clickhouse-webhooks-events, .gemini/skills/clickhouse-webhooks-events, .github/skills/clickhouse-webhooks-events and .opencode/skills/clickhouse-webhooks-events in your project.

What does Clickhouse Webhooks Events need to run?

Going by SKILL.md and its folder, Clickhouse Webhooks Events needs the command-line tools its instructions call (curl). Its frontmatter pre-approves these tools: Read, Write, Edit, Bash(curl:*). Compatibility (from SKILL.md): Designed for Claude Code.

Does Clickhouse Webhooks Events access the network?

SKILL.md names 1 domain. As links in the text: clickhouse.com. This is read from the text; nothing was executed.

Is Clickhouse Webhooks Events safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Clickhouse Webhooks Events use?

Clickhouse Webhooks Events is published under the MIT licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Clickhouse Webhooks Events use?

About 1.7k tokens (SKILL.md is roughly 6.6k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 1.3k tokens, read only when the agent opens those files.

What are the alternatives to Clickhouse Webhooks Events?

Skills that share tags, products or a category with Clickhouse Webhooks Events: Querying Tophog (PostHog/posthog, 40k stars), AWS Storage (aws/agent-toolkit-for-aws, 2.8k stars), Opensourcefaq (digoal/blog, 8.6k stars) and Monitoring Ingestion Pipeline (PostHog/posthog, 40k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Clickhouse Webhooks Events?

jeremylongshore (a GitHub user) maintains it in jeremylongshore/tons-of-skills-marketplace, which has 2,827 GitHub stars. The repository holds 3,342 skills in this directory. The repository was last updated on October 10, 2026.

Source: jeremylongshore/tons-of-skills-marketplace on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.