Kafka Development

Kafka Development

This skill provides best practices for Apache Kafka event streaming and distributed messaging systems. Apply these guidelines when building Kafka-based applications.

Core Principles

• Kafka is a distributed event streaming platform for high-throughput, fault-tolerant messaging
• Unlike traditional pub/sub, Kafka uses a pull model - consumers pull messages from partitions
• Design for scalability, durability, and exactly-once semantics where needed
• Leave NO todos, placeholders, or missing pieces in the implementation

Workflow: Setting Up a Kafka Producer-Consumer Pipeline

1. Define the topic — choose a descriptive name, set partition count based on expected consumer parallelism, and configure retention and replication factor.
2. Design the message schema — register an Avro or JSON schema in Schema Registry; ensure backward compatibility from the start.
3. Implement the producer — configure acks=all, enable idempotence, select a partition key that distributes evenly, and add error handling with retry logic.
4. Implement the consumer — set enable.auto.commit=false, pick an appropriate auto.offset.reset policy, process messages idempotently, and commit offsets only after successful processing.
5. Add observability — instrument producer send-rate, consumer lag, and broker under-replicated-partitions; propagate trace context in message headers.
6. Test end-to-end — use Testcontainers or an embedded Kafka broker to verify the full produce-consume-commit cycle, including failure and rebalance scenarios.
7. Deploy and monitor — roll out with lag alerts, dead-letter-topic routing for persistent failures, and dashboards for key broker and client metrics.

Architecture Overview

Core Components

• Topics: Categories/feeds for organizing messages
• Partitions: Ordered, immutable sequences within topics enabling parallelism
• Producers: Clients that publish messages to topics
• Consumers: Clients that read messages from topics
• Consumer Groups: Coordinate consumption across multiple consumers
• Brokers: Kafka servers that store data and serve clients

Key Concepts

• Messages are appended to partitions in order
• Each message has an offset - a unique sequential ID within the partition
• Consumers maintain their own cursor (offset) and can read streams repeatedly
• Partitions are distributed across brokers for scalability

Topic Design

Partitioning Strategy

• Use partition keys to place related events in the same partition
• Messages with the same key always go to the same partition
• This ensures ordering for related events
• Choose keys carefully - uneven distribution causes hot partitions

Partition Count

• More partitions = more parallelism but more overhead
• Consider: expected throughput, consumer count, broker resources
• Start with number of consumers you expect to run concurrently
• Partitions can be increased but not decreased

Topic Configuration

• retention.ms: How long to keep messages (default 7 days)
• retention.bytes: Maximum size per partition
• cleanup.policy: delete (remove old) or compact (keep latest per key)
• min.insync.replicas: Minimum replicas that must acknowledge

Producer Best Practices

Reliability Settings

acks=all               # Wait for all replicas to acknowledge
retries=MAX_INT        # Retry on transient failures
enable.idempotence=true # Prevent duplicate messages on retry

Performance Tuning

• batch.size: Accumulate messages before sending (default 16KB)
• linger.ms: Wait time for batching (0 = send immediately)
• buffer.memory: Total memory for buffering unsent messages
• compression.type: gzip, snappy, lz4, or zstd for bandwidth savings

Error Handling

• Implement retry logic with exponential backoff
• Handle retriable vs non-retriable exceptions differently
• Log and alert on send failures
• Consider dead letter topics for messages that fail repeatedly

Example: Java Producer with Idempotence and Error Handling

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy");

try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
    ProducerRecord<String, String> record =
        new ProducerRecord<>("orders", "order-123", "{\"item\":\"widget\",\"qty\":5}");

    producer.send(record, (metadata, exception) -> {
        if (exception != null) {
            log.error("Send failed for key=order-123", exception);
            // Route to dead-letter topic or alert
        } else {
            log.info("Delivered to {}-{} offset {}",
                metadata.topic(), metadata.partition(), metadata.offset());
        }
    });
}

Partitioner

• Default: hash of key determines partition (null key = round-robin)
• Custom partitioners for specific routing needs
• Ensure even distribution to avoid hot partitions

Consumer Best Practices

Offset Management

• Consumers track which messages they've processed via offsets
• auto.offset.reset: earliest (start from beginning) or latest (only new messages)
• Commit offsets after successful processing, not before
• Use enable.auto.commit=false for exactly-once semantics

Consumer Groups

• Consumers in a group share partitions (each partition to one consumer)
• More consumers than partitions = some consumers idle
• Group rebalancing occurs when consumers join/leave
• Use group.instance.id for static membership to reduce rebalances

Processing Patterns

• Process messages in order within a partition
• Handle out-of-order messages across partitions if needed
• Implement idempotent processing for at-least-once delivery
• Consider transactional processing for exactly-once

Example: Java Consumer with Manual Offset Commit

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-processing-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(Collections.singletonList("orders"));

    while (running) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> record : records) {
            try {
                processOrder(record.key(), record.value());
            } catch (Exception e) {
                log.error("Failed to process offset={} key={}", record.offset(), record.key(), e);
                publishToDeadLetterTopic(record, e);
            }
        }
        consumer.commitSync(); // Commit only after successful processing
    }
}

Timeouts and Failures

• Implement processing timeout to isolate slow events
• When timeout occurs, set event aside and continue to next message
• Maintain overall system performance over processing every single event
• Use dead letter queues for messages failing all retries

Error Handling and Retry

Retry Strategy

• Allow multiple runtime retries per processing attempt
• Example: 3 runtime retries per redrive, maximum 5 redrives = 15 total retries
• Runtime retries typically cover 99% of failures
• After exhausting retries, route to dead letter queue