Cloudflare Queues

Cloudflare Queues

Status: Production Ready ✅ | Last Verified: 2025-12-27

Dependencies: cloudflare-worker-base (for Worker setup)

Contents: Quick Start • Critical Rules • Top Errors • Use Cases • When to Load References • Limits

Quick Start (10 Minutes)

1. Create Queue

bunx wrangler queues create my-queue
bunx wrangler queues list

2. Producer (Send Messages)

wrangler.jsonc:

{
  "name": "my-producer",
  "main": "src/index.ts",
  "queues": {
    "producers": [
      {
        "binding": "MY_QUEUE",
        "queue": "my-queue"
      }
    ]
  }
}

src/index.ts:

import { Hono } from 'hono';

type Bindings = {
  MY_QUEUE: Queue;
};

const app = new Hono<{ Bindings: Bindings }>();

app.post('/send', async (c) => {
  await c.env.MY_QUEUE.send({
    userId: '123',
    action: 'process-order',
    timestamp: Date.now(),
  });

  return c.json({ status: 'queued' });
});

export default app;

3. Consumer (Process Messages)

wrangler.jsonc:

{
  "name": "my-consumer",
  "main": "src/consumer.ts",
  "queues": {
    "consumers": [
      {
        "queue": "my-queue",
        "max_batch_size": 10,
        "max_retries": 3,
        "dead_letter_queue": "my-dlq"
      }
    ]
  }
}

src/consumer.ts:

import type { MessageBatch } from '@cloudflare/workers-types';

export default {
  async queue(batch: MessageBatch): Promise<void> {
    for (const message of batch.messages) {
      console.log('Processing:', message.body);
      // Your logic here
    }
    // Implicit ack: returning successfully acknowledges all messages
  },
};

Deploy:

bunx wrangler deploy

Load: references/setup-guide.md for complete 6-step setup with DLQ configuration

Critical Rules

Always Do ✅

1. Configure Dead Letter Queue for production queues
2. Use explicit ack() for non-idempotent operations (DB writes, API calls)
3. Validate message size before sending (<128 KB)
4. Use sendBatch() for multiple messages (more efficient)
5. Implement exponential backoff for retries
6. Set appropriate batch settings based on workload
7. Monitor queue backlog and consumer errors
8. Use ctx.waitUntil() for async cleanup in consumers
9. Handle errors gracefully - log, alert, retry
10. Let concurrency auto-scale (don't set max_concurrency unless needed)

Never Do ❌

1. Never assume message ordering - not guaranteed FIFO
2. Never rely on implicit ack for non-idempotent ops - use explicit ack()
3. Never send messages >128 KB - will fail
4. Never delete queues with active messages - data loss
5. Never skip DLQ configuration in production
6. Never exceed 5000 msg/s per queue - rate limit error
7. Never process messages synchronously in loop - use Promise.all()
8. Never ignore message.attempts - use for backoff logic
9. Never set max_concurrency=1 unless you have a very specific reason
10. Never forget to ack() in explicit acknowledgement patterns

Top 3 Critical Errors

Error #1: Message Too Large

Problem: Message exceeds 128 KB limit

Solution: Store large data in R2, send reference

// ❌ Wrong
await env.MY_QUEUE.send({ data: largeArray }); // >128 KB fails

// ✅ Correct
const message = { data: largeArray };
const size = new TextEncoder().encode(JSON.stringify(message)).length;

if (size > 128000) {
  const key = `messages/${crypto.randomUUID()}.json`;
  await env.MY_BUCKET.put(key, JSON.stringify(message));
  await env.MY_QUEUE.send({ type: 'large-message', r2Key: key });
} else {
  await env.MY_QUEUE.send(message);
}

Error #2: Throughput Exceeded

Problem: Exceeding 5000 messages/second per queue

Solution: Use sendBatch() and rate limiting

// ❌ Wrong
for (let i = 0; i < 10000; i++) {
  await env.MY_QUEUE.send({ id: i }); // Too fast!
}

// ✅ Correct
const messages = Array.from({ length: 10000 }, (_, i) => ({
  body: { id: i },
}));

// Send in batches of 100
for (let i = 0; i < messages.length; i += 100) {
  await env.MY_QUEUE.sendBatch(messages.slice(i, i + 100));
}

Error #3: Entire Batch Retried When One Message Fails

Problem: Single message failure causes all messages to retry

Solution: Use explicit acknowledgement

// ❌ Wrong - implicit ack
export default {
  async queue(batch: MessageBatch, env: Env): Promise<void> {
    for (const message of batch.messages) {
      await env.DB.prepare('INSERT INTO orders VALUES (?, ?)').bind(
        message.body.id,
        message.body.amount
      ).run();
    }
    // If any fails, ALL retry!
  },
};

// ✅ Correct - explicit ack
export default {
  async queue(batch: MessageBatch, env: Env): Promise<void> {
    for (const message of batch.messages) {
      try {
        await env.DB.prepare('INSERT INTO orders VALUES (?, ?)').bind(
          message.body.id,
          message.body.amount
        ).run();

        message.ack(); // Only ack on success
      } catch (error) {
        console.error(`Failed: ${message.id}`, error);
        // Don't ack - will retry independently
      }
    }
  },
};

Load references/error-catalog.md for all 10 errors including DLQ configuration, auto-scaling issues, message deletion prevention, and detailed solutions.

Common Use Cases

Use Case 1: Basic Message Queue

When: Simple async job processing (emails, notifications)

Quick Pattern:

// Producer
await env.MY_QUEUE.send({ type: 'email', to: 'user@example.com' });

// Consumer (implicit ack - for idempotent operations)
export default {
  async queue(batch: MessageBatch): Promise<void> {
    for (const message of batch.messages) {
      await sendEmail(message.body.to, message.body.content);
    }
  },
};

Load: templates/queues-producer.ts + templates/queues-consumer-basic.ts

Use Case 2: Database Writes (Non-Idempotent)

When: Writing to database, must avoid duplicates

Load: templates/queues-consumer-explicit-ack.ts + references/consumer-api.md

Use Case 3: Retry with Exponential Backoff

When: Calling rate-limited APIs, temporary failures

Load: templates/queues-retry-with-delay.ts + references/error-catalog.md (Error #2, #3)

Use Case 4: Dead Letter Queue Pattern

When: Production systems, need to capture permanently failed messages

Load: templates/queues-dlq-pattern.ts + references/setup-guide.md (Step 4)

Use Case 5: High Throughput Processing

When: Processing thousands of messages per second

Quick Pattern:

{
  "queues": {
    "consumers": [{
      "queue": "my-queue",
      "max_batch_size": 100,        // Large batches
      "max_batch_timeout": 5,       // Fast processing
      "max_concurrency": null       // Auto-scale
    }]
  }
}

Load: references/best-practices.md → Optimizing Throughput

When to Load References

Load references/setup-guide.md when:

• User needs complete setup walkthrough (queue → producer → consumer → DLQ)
• First time setting up Cloudflare Queues
• Need production configuration examples
• Want complete end-to-end example

Load references/error-catalog.md when:

• Encountering any of the 10 documented errors
• Troubleshooting queue issues
• Messages not being delivered/processed
• Need prevention checklist

Load references/producer-api.md when:

• Need complete producer API reference
• Using send() or sendBatch() methods
• Need message format specifications
• Handling large messages or batches

Load references/consumer-api.md when:

• Need complete consumer API reference
• Using explicit ack(), retry(), or retryAll()
• Need batch processing patterns
• Implementing complex consumer logic

Load references/best-practices.md when:

• Optimizing queue performance
• Production deployment guidance
• Monitoring and observability setup
• Security and reliability patterns

Load references/wrangler-commands.md when:

• Need CLI commands reference
• Managing queues (create, delete, list)
• Controlling delivery (pause, resume)
• Debugging queue issues
• Real-time monitoring and performance analysis

Load references/typescript-types.md when:

• Need complete TypeScript type definitions
• Working with Queue, MessageBatch, or Message interfaces
• Implementing type-safe producers or consumers
• Using generic types for message bodies
• Need type guards for message validation

Load references/production-checklist.md when:

• Preparing for production deployment
• Need pre-deployment verification checklist
• Want detailed explanations of production best practices
• Setting up monitoring, DLQ, or error handling
• Planning load testing or security review

Load references/pull-consumers.md when:

• Need to consume messages from non-Workers environments
• Integrating with existing backend services (Node.js, Python, Go)
• Implementing pull-based polling instead of push-based consumption
• Working with containerized or legacy applications

Load references/http-publishing.md when:

• Publishing messages from external systems via HTTP
• Integrating webhooks from third-party services
• Need to send messages without deploying Workers
• Implementing CI/CD pipeline notifications

Load references/r2-event-integration.md when:

• Triggering queue messages on R2 bucket events
• Implementing automated image/document processing
• Setting up event-driven data pipelines
• Need R2 object upload/delete notifications