Handling Payment Webhooks Reliably: Idempotency and Retries

Payment webhook idempotency is the difference between a wallet credited once and a wallet credited three times. Every payment provider I have integrated with — Stripe, GoCardless, a couple of local PSPs — retries a webhook until it gets a 2xx, and they make no promise about delivery order. So a `charge.succeeded` you already processed arrives again because your first 200 was slow, and a stale `payment.pending` lands after the `payment.paid` that superseded it. A naive handler that just reads the payload and mutates a balance will double-credit on the retry and overwrite a good status with the old one. The fix is two things working together: dedupe every delivery by the provider's event id behind a database unique constraint, and model the payment as a state machine that refuses to move backwards. Persist the raw event, return 200 fast, do the real work in a queued job.

Why does the same webhook arrive more than once?

Providers treat your endpoint as unreliable, which is correct, so they retry. Stripe will redeliver an event for up to three days with exponential backoff if it does not see a 2xx in time. The catch is that a retry is also triggered when your handler did succeed but took too long to answer — your worker credited the wallet, the request timed out at the load balancer, the provider never saw the 200, and it sends the exact same event again. Same `evt_` id, same body. If your only protection is 'have I seen this balance change recently', you lose. The id is the stable thing to key on, not the effect.

Out-of-order is the second half. Webhook A (`pending`) and webhook B (`paid`) are dispatched milliseconds apart; B's HTTP request wins the race or A's gets retried after a transient failure. Now `pending` arrives last and a status-overwriting handler walks the payment backwards. Idempotency alone does not save you here — both events are genuinely distinct deliveries with distinct ids. You need ordering logic on top.

How do I dedupe by event id under concurrent retries?

Put a unique constraint on the provider event id and let the database be the arbiter. Application-level 'select then insert' checks lose to concurrency — two retries hitting two workers both read 'not seen', both proceed. A unique index is the only check that holds when deliveries land at the same instant. The migration is one line that does the heavy lifting.

Schema::create('webhook_events', function (Blueprint $table) {
    $table->id();
    $table->string('provider');           // 'stripe', 'gocardless', ...
    $table->string('event_id');           // evt_1Nx... from the provider
    $table->string('type');               // 'charge.succeeded'
    $table->json('payload');              // the raw, verified body
    $table->timestamp('processed_at')->nullable();
    $table->timestamps();

    // This composite unique key is the whole game. A duplicate delivery
    // for the same provider hits it and is rejected at the DB layer,
    // even if two retries race on two queue workers.
    $table->unique(['provider', 'event_id']);
});

In the controller I do a plain `create()` on that pair and lean on the unique index, not a `firstOrCreate` — a `firstOrCreate` still does a select-then-insert and loses the same race I am trying to close. If a concurrent insert wins, the second `create` throws a `QueryException` with SQLSTATE `23000` for the duplicate key, and I treat that as 'already received' rather than an error. Catch it narrowly and return a 2xx — the provider has its acknowledgement and stops retrying. This is the persist-and-acknowledge step; verifying the signature happens just before it, which I cover in the webhook signature verification post, and the routing of every provider through one entry point is in centralizing payment webhooks in a Laravel controller.

public function handle(Request $request, string $provider)
{
    // Signature verification happens here, before anything is trusted.
    $event = $this->verify($provider, $request);

    try {
        $record = WebhookEvent::create([
            'provider'  => $provider,
            'event_id'  => $event['id'],
            'type'      => $event['type'],
            'payload'   => $event,
        ]);
    } catch (\Illuminate\Database\QueryException $e) {
        // SQLSTATE 23000 = integrity constraint violation (duplicate event_id).
        // We have seen this delivery. Acknowledge and stop the retries.
        if ($e->getCode() === '23000') {
            return response()->noContent(); // 204, a 2xx — provider is happy
        }
        throw $e;
    }

    // Return fast, do the real work off the request thread.
    ProcessWebhookEvent::dispatch($record);

    return response()->noContent();
}

Why persist first and process in a queue?

Two reasons. First, latency: the provider expects a 2xx within seconds, and a Stripe timeout is treated as failure and triggers a retry storm. If you credit a wallet, call a ledger service, and send a receipt email all inside the request, you will blow that budget and create the exact duplicate-delivery problem you are trying to avoid. Persist the raw event, dispatch a job, return. Second, durability: once the row is committed, the event is safe even if the worker crashes mid-process — the job retries from a stored record, not from an HTTP request you can never get back. The queue plus the unique constraint is what gives you at-most-once effects on the side that matters.

Run those workers under a process supervisor so a crash does not silently stop processing; I walk through that setup in running Laravel queue workers with Supervisor in production. Inside the job, wrap the state change in a transaction and mark `processed_at` so a re-queued job is a no-op on the effect, not just the receipt.

How do I stop a late event from undoing a finished payment?

Model the payment status as a state machine and define which transitions are legal. A payment moves forward — `pending` to `paid` to `refunded` — and a webhook that asks to move it backward gets ignored, not applied. When the stale `pending` lands after `paid`, the job looks at the current state, sees that `pending` is not a legal successor of `paid`, logs it, and drops the transition. No overwrite, no corruption.

• Define allowed transitions explicitly: from `pending` you may go to `paid`, `failed`, or `canceled`; from `paid` only to `refunded`. Anything else is a no-op.
• Compare against the persisted current state inside the transaction, not against whatever you read at the top of the request — the row may have changed.
• Where the provider includes one, prefer an event timestamp or sequence number over your own arrival order to decide what is newer.
• Treat an illegal transition as expected, not exceptional: log it at info level and return success so the provider does not retry a delivery you have deliberately ignored.

private const TRANSITIONS = [
    'pending' => ['paid', 'failed', 'canceled'],
    'paid'    => ['refunded'],
    'failed'  => [],
    'refunded'=> [],
];

public function handle(): void
{
    DB::transaction(function () {
        // Reload inside the transaction; lock the row against concurrent jobs.
        $payment = Payment::whereKey($this->paymentId())->lockForUpdate()->first();
        $next    = $this->mapEventToStatus($this->event->type);

        if (! in_array($next, self::TRANSITIONS[$payment->status] ?? [], true)) {
            // Late or duplicate transition (e.g. a 'pending' after 'paid').
            Log::info('Ignored backward webhook transition', [
                'payment' => $payment->id,
                'from'    => $payment->status,
                'to'      => $next,
                'event'   => $this->event->event_id,
            ]);
            return; // no-op, but the job succeeds
        }

        $payment->update(['status' => $next]);
        $this->event->update(['processed_at' => now()]);
    });
}

Treat every webhook as something you have already seen and something that arrived in the wrong order. Design for both and the happy path takes care of itself.Md Raihan Hasan

What about replay and knowing it actually worked?

Keeping the raw, verified payload in that `webhook_events` row is not just an audit nicety — it is your recovery mechanism. When a downstream service was down and a batch of events failed, you re-dispatch `ProcessWebhookEvent` from the stored rows; you do not ask the provider to resend, and you cannot, because the original request is long gone. Because every job is idempotent against the payment state, replaying the whole table is safe: events already applied become no-ops, only the genuinely-unprocessed ones take effect. Log every event id, type, and the transition decision so that when finance asks why a payment shows `paid`, you can point at the exact delivery that moved it there.

None of this is exotic. A unique constraint on the provider event id, a persist-then-queue handler that answers in milliseconds, and a state machine that refuses to walk backwards — three small pieces that turn an unreliable, out-of-order stream of retries into exactly-once effects on the only thing that matters, the money. Build it once as a base handler, verify the signature before you trust a byte, and every provider you add later inherits the same guarantees. The first time a provider hammers you with the same `charge.succeeded` four times in a minute, you will be glad the answer is a 200 and a no-op instead of a support ticket about a triple-charged customer.