1. The Distributed Webhook Failure Problem

In B2B SaaS platforms, CPaaS engines, and fintech systems, emitting webhooks to notify client endpoints of asynchronous events (e.g., payment completed, SMS delivered) is mission-critical. However, relying on a standard HTTP POST request inside your core application flow introduces severe failure modes:

  • Network Partitions: If the client’s endpoint is down or returning a 504 Gateway Timeout, your main database transaction may block or fail.
  • Dual-Write Inconsistencies: If you save a state change in your PostgreSQL database and immediately trigger an external HTTP call, a crash between the two actions results in lost notifications.

2. Core Architecture: The Transactional Outbox Pattern

To solve dual-write inconsistencies, systems use the Transactional Outbox Pattern. Instead of calling external APIs directly inside business logic, events are written to an outbox table within the exact same database transaction as the state change. A separate background worker polls the outbox table and dispatches payloads reliably.

3. Database Schema Design for Outbox Queues

CREATE TABLE webhook_outbox (
    id BIGSERIAL PRIMARY KEY,
    tenant_id UUID NOT NULL,
    event_type VARCHAR(100) NOT NULL,
    target_url TEXT NOT NULL,
    payload JSONB NOT NULL,
    status VARCHAR(32) DEFAULT 'PENDING', -- PENDING, PROCESSING, COMPLETED, FAILED
    attempt_count INT DEFAULT 0,
    max_attempts INT DEFAULT 5,
    next_retry_at_epoch BIGINT NOT NULL,
    created_at_epoch BIGINT NOT NULL
);

-- Index for background worker polling performance
CREATE INDEX idx_outbox_polling ON webhook_outbox (status, next_retry_at_epoch) WHERE status = 'PENDING';

4. Background Worker with Exponential Backoff

When a webhook delivery fails due to a network timeout or a 5xx response, the worker must not hammer the target server continuously. It should calculate an exponential backoff interval with jitter to prevent thundering herd problems on recovering servers.

Node.js Worker Processing Snippet:

const axios = require('axios');
const { Pool } = require('pg');
const pool = new Pool();

async function processWebhookOutbox() {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    // Fetch batch of pending webhooks ready for delivery
    const now = Math.floor(Date.now() / 1000);
    const result = await client.query(
      `SELECT id, target_url, payload, attempt_count, max_attempts 
       FROM webhook_outbox 
       WHERE status = 'PENDING' AND next_retry_at_epoch <= $1 
       LIMIT 50 FOR UPDATE SKIP LOCKED`,
      [now]
    );

    for (const row of result.rows) {
      try {
        // Dispatch webhook with cryptographic signature header
        await axios.post(row.target_url, row.payload, {
          timeout: 5000,
          headers: { 'X-TecqMate-Signature': generateSignature(row.payload) }
        });

        // Mark as completed
        await client.query(`UPDATE webhook_outbox SET status = 'COMPLETED' WHERE id = $1`, [row.id]);
      } catch (err) {
        const nextAttempt = row.attempt_count + 1;
        if (nextAttempt >= row.max_attempts) {
          // Move to dead-letter state after max retries
          await client.query(`UPDATE webhook_outbox SET status = 'FAILED', attempt_count = $1 WHERE id = $2`, [nextAttempt, row.id]);
        } else {
          // Calculate exponential backoff: 2^attempt * 10 seconds + random jitter
          const backoffSeconds = Math.pow(2, nextAttempt) * 10 + Math.floor(Math.random() * 5);
          const nextRetry = now + backoffSeconds;
          
          await client.query(
            `UPDATE webhook_outbox SET attempt_count = $1, next_retry_at_epoch = $2 WHERE id = $3`,
            [nextAttempt, nextRetry, row.id]
          );
        }
      }
    }

    await client.query('COMMIT');
  } catch (error) {
    await client.query('ROLLBACK');
    console.error('Outbox worker cycle error:', error.message);
  } finally {
    client.release();
  }
}

5. Production Best Practices

  1. Cryptographic Signatures: Always sign webhook payloads using HMAC-SHA256 (e.g., X-Signature: sha256=...) so recipient clients can cryptographically verify that requests originated from your platform and haven’t been tampered with.
  2. Idempotency on the Receiving End: Remind your B2B clients in your documentation that webhooks can occasionally be delivered more than once due to network retries, so their ingestion endpoints must be idempotent.
  3. Dead-Letter Monitoring: Expose internal monitoring dashboards or alerts for webhooks entering the FAILED state after maximum retry exhaustion.