Post-mortem: The Stripe webhook that doubled our charges
How a missing idempotency check in our Stripe webhook handler led to a $19K overcharge, and the four guardrails we now wrap every webhook in.
The Dench Team
·8 min read
Post-mortem: The Stripe webhook
On April 12, 2026 at 14:42 UTC, a subset of customers received a second invoice for a billing period they'd already paid for. Total impact: 37 customers, $19,400 in mistaken charges, all refunded within 48 hours.
Root cause: a Stripe retry of the invoice.payment_succeeded event arrived while the original handler was still in-flight. Both handlers ran to completion, neither saw the other's row, and both inserted a billing-period record.
Fixes:
stripeEventReceiptstable. Every webhook now writes an idempotency receipt before doing any work. Re-deliveries see the receipt and short-circuit.- Synchronous read-modify-write. Inside a Convex mutation, the receipt + the side-effecting row write happen in the same transaction.
- Out-of-band alerts. PagerDuty rings when the receipt table sees more than 10 dupes in 5 minutes.
- Refund automation. Detected dupes auto-refund within 10 minutes instead of waiting for human review.
No customer churned. The fix took 4 hours; the post-mortem and follow-ups took another 12. Worth every minute.
