	In Browser
	StumbleUpon
	del.icio.us
	Google
	Google Buzz
	reddit
	LinkedIn

	Facebook
	Twitter
	Linkedin
	E-Mail

Messaging > Apache Kafka > Kafka Dead Letter Queue

Kafka Dead Letter Queue

Author: Venkata Sudhakar

A dead letter queue (DLQ) is a separate Kafka topic where messages that cannot be processed successfully are sent instead of blocking the consumer. Without a DLQ, a single bad message (called a poison pill) causes your consumer to retry forever, blocking all processing on that partition. With a DLQ, the bad message is moved aside after a set number of retries so processing continues for good messages, and the failed messages can be inspected and reprocessed later.

In Spring Kafka, DefaultErrorHandler combined with DeadLetterPublishingRecoverer implements this automatically. You configure how many retries to attempt and how long to wait between them. After retries are exhausted, the recoverer publishes the failed message to a topic named originalTopic.DLT with headers containing the exception message, stack trace, and original partition and offset so you can diagnose what went wrong.

The below example shows how to configure a dead letter topic in a Spring Boot Kafka consumer so bad messages are routed to orders.DLT after 3 failed attempts.

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.util.backoff.FixedBackOff;

@Configuration
public class KafkaErrorConfig {

@Bean
    public DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> kafkaTemplate) {
        // After retries are exhausted, publish failed message to orders.DLT
        DeadLetterPublishingRecoverer recoverer =
            new DeadLetterPublishingRecoverer(kafkaTemplate);

// 2 retries, 1 second apart (3 total attempts)
        FixedBackOff backOff = new FixedBackOff(1000L, 2L);

DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);

// Skip retries for data format errors - go straight to DLT
        handler.addNotRetryableExceptions(IllegalArgumentException.class);

return handler;
    }
}

The consumer service that handles messages and the DLT monitor,

It gives the following output when a bad message arrives,

Processed: {"orderId":"ORD-1001","amount":99.99}

[Attempt 1] RuntimeException: Corrupt order: CORRUPT-ORD-999
[Attempt 2] RuntimeException: Corrupt order: CORRUPT-ORD-999
[Attempts exhausted] Publishing to orders.DLT

ALERT - failed message in DLT: CORRUPT-ORD-999

Processed: {"orderId":"ORD-1002","amount":49.99}

The DLT message carries headers added automatically by Spring Kafka: kafka_dlt-exception-message holds the error text, kafka_dlt-original-topic, kafka_dlt-original-partition, and kafka_dlt-original-offset pinpoint exactly where the message came from. These headers make manual reprocessing or debugging straightforward without needing to search the original topic.

Send your comments, suggestions or queries regarding this site to [email protected].