Webhook retries that actually land

Outbound webhooks fail for boring reasons: cold starts, 5xx spikes, DNS blips, brief deploys that return 502. Treating a single POST as success is how CRM state silently drifts from your live chat reality.

This guide is for backend engineers consuming CanWeChat’s signed webhook relay—or anyone designing their own relay. You will get a verification snippet, an idempotency pattern, and a debugging checklist tied to how CanWeChat actually delivers events.

Quick answer

• Every CanWeChat outbound payload is HMAC-SHA256 signed over timestamp + raw body.
• Failed deliveries retry from a Cloudflare Queue; duplicates are normal.
• Key off event id (and X-CanWeChat-Id) before side effects.
• Return 2xx only after durable upsert; otherwise we will retry.
• Delivery history stores status, attempt number, and a truncated response body for debugging.

Why fire-and-forget fails

A visitor message path that awaits your CRM’s HTTP round-trip couples product UX to someone else’s uptime. CanWeChat accepts the visitor message, persists it, then enqueues outbound work. If your endpoint is down, the queue retries. That is the correct failure mode—as long as your handler is idempotent.

The signature contract

Headers: X-CanWeChat-Id, X-CanWeChat-Timestamp, and X-CanWeChat-Signature with a v1= hex digest. Compute HMAC-SHA256 of `${timestamp}.${rawBody}` with the relay secret. Compare with a timing-safe equal. Reject timestamps outside your skew window.

import { createHmac, timingSafeEqual } from "node:crypto";

type VerifyInput = {
  rawBody: string;
  timestamp: string;
  signatureHeader: string;
  secret: string;
  maxSkewSec?: number;
};

export function verifyRelay(input: VerifyInput): boolean {
  const skew = input.maxSkewSec ?? 300;
  const ts = Number(input.timestamp);
  if (!Number.isFinite(ts)) return false;
  if (Math.abs(Date.now() / 1000 - ts) > skew) return false;

  const expected =
    "v1=" +
    createHmac("sha256", input.secret)
      .update(`${input.timestamp}.${input.rawBody}`)
      .digest("hex");

  const a = Buffer.from(expected);
  const b = Buffer.from(input.signatureHeader);
  return a.length === b.length && timingSafeEqual(a, b);
}

Idempotent handler pattern

1. Read raw body; verify signature and timestamp.
2. Parse JSON; read event.id.
3. BEGIN transaction / upsert event row with unique id.
4. If the row already existed, return 204 immediately.
5. Else apply side effects (CRM note, email, ticket) that can tolerate replay or are guarded by the same id.
6. COMMIT; return 2xx.

// Pseudocode — adapt to your DB
async function handle(rawBody, headers) {
  if (!verifyRelay({
    rawBody,
    timestamp: headers["x-canwechat-timestamp"],
    signatureHeader: headers["x-canwechat-signature"],
    secret: process.env.CANWECHAT_WEBHOOK_SECRET,
  })) {
    return { status: 401 };
  }

  const event = JSON.parse(rawBody);
  const inserted = await db.insertIgnore("canwechat_events", {
    id: event.id,
    type: event.type,
    payload: event,
  });

  if (!inserted) return { status: 204 };

  if (event.type === "message.created") {
    await crm.appendNote({
      idempotencyKey: event.id,
      conversationId: event.conversation.id,
      body: event.message.body,
      email: event.visitor?.email,
    });
  }

  return { status: 204 };
}

What CanWeChat retries

Non-2xx responses and transport failures re-enter a Cloudflare Queue. The dashboard delivery log shows event type, delivery status, HTTP code, attempt number, timestamp, and a bounded response snippet—enough to see “connection refused” or a framework error page without storing megabytes of HTML.

Think of the queue as a shock absorber between visitor UX and your CRM. The widget send path should not wait on HubSpot’s cold start. Likewise, a brief CanWeChat Worker deploy should not drop a message that already landed in D1—outbound work resumes when your endpoint recovers.

Receiver guidelines (not a vendor SLA)

• Aim to acknowledge quickly; do heavy CRM work after upsert if needed.
• Do not return 200 before persistence—retries will duplicate side effects.
• Prefer 401/403 for bad signatures (fix config) vs 500 (retryable).
• Watch for systematic 413/timeout if you buffer entire bodies in memory on huge payloads (CanWeChat message bodies are modest; still be deliberate).

Status code semantics that help everyone

• 2xx: event accepted and durably recorded (or already seen).
• 401/403: signature or auth problem—fix secrets; retries will keep failing until you do.
• 404: wrong path after a deploy—update the relay URL in the dashboard.
• 408/429/5xx: transient—expect another attempt from the queue.
• 3xx: do not design receivers that require following redirects for webhook POSTs; configure the final HTTPS URL.

Local test harness sketch

Before you point production CanWeChat at a tunnel URL, rehearse verification with a known body, timestamp, and secret. A ten-line script that prints pass/fail beats discovering skew bugs after a launch email.

# Generate a signature for a fixture body (Node one-liner sketch)
node -e '
const { createHmac } = require("crypto");
const body = JSON.stringify({ id: "evt_test", type: "message.created" });
const ts = Math.floor(Date.now() / 1000).toString();
const secret = process.env.CANWECHAT_WEBHOOK_SECRET;
const sig = "v1=" + createHmac("sha256", secret).update(ts + "." + body).digest("hex");
console.log({ ts, sig, body });
'

Events you will typically see

• conversation.created — a visitor started a new thread from the widget.
• message.created — a visitor or agent message landed (check senderType / source).
• Future event types may appear; ignore unknown types after storing the id so deploys stay forward-compatible.

Debugging checklist

1. Confirm the relay URL in the dashboard matches the deployed route.
2. Compare signing secrets after any rotation.
3. Log whether verification fails before JSON parse.
4. Inspect delivery history for HTTP codes and response snippets.
5. Prove idempotency by replaying a test event and checking for a single CRM row.
6. Ensure your endpoint does not follow open redirects or hit internal IPs if you proxy—SSRF hygiene matters on both sides.

The CanWeChat angle

Retries are part of the product contract, not optional polish. Queues keep Slack/Discord/Telegram and customer webhooks off the visitor’s send path. The widget stays a small Shadow DOM IIFE; the messy distributed parts stay on Workers. Configure the URL in the dashboard—there is no public REST API to manage the relay, by design.

Limitations

• Retries improve delivery; they do not create a public uptime guarantee.
• If your handler is not idempotent, retries will amplify bugs.
• We truncate response snippets—deep debugging still needs your own logs.
• Secret rotation requires coordinated updates; old signatures will 401 until fixed.

When something looks “stuck,” open the dashboard delivery history first. Most production incidents are wrong URLs, rotated secrets, or handlers that return 200 before the database commit—not mysterious Cloudflare Queue bugs. Fix the receiver contract, then send another test event.