Paid Once, Recorded Once: Engineering Payment Flows That Withstand Repetition

At 08:42 on a Tuesday morning, a customer pays an invoice. At 08:43 the payment provider sends a webhook. At 08:43 and four seconds it sends the same webhook again, because the first response was slow. At 08:44 the customer refreshes the page. By 09:15 support has two tickets, finance has one payment and the order system has three events. Nobody has done anything obviously wrong, yet the product is now asking an awkward question: was this customer charged once or twice?

That question is not a support inconvenience. In a UK business product it is a matter of trust, regulatory hygiene and, eventually, cash. Payment flows need to be engineered on the assumption that messages repeat, arrive late and occasionally appear in the wrong order. The alternative is a system that is probably fine until the first busy Tuesday.

The default assumption is wrong

Most payment integrations begin with an implicit belief that a request produces one response and one side effect. Real systems do not behave that way. Networks time out. Providers retry. Browsers resend form submissions. Mobile connections drop between the authorisation and the confirmation. A payment provider may deliver the same event more than once precisely because it cannot be certain the first delivery was received.

This is not a defect in the provider. It is the normal behaviour of distributed systems operating over unreliable channels. The engineering response is not to hope for exactly-once delivery. It is to make the receiving system tolerant of repetition, clear about state and able to prove what happened after the fact.

Make idempotency a property of the operation

An idempotency key is often described as a request header. In practice it is a contract about an operation. The key should be generated by the client or by the initiating service, scoped to the business action and stored alongside the outcome of that action. If the same key arrives again, the system returns the original result rather than performing the work twice.

Several design choices matter. First, do not derive the key from mutable details such as the amount and the current date. A customer correcting a typo in their reference should not create a second charge simply because the payload changed. Second, enforce the key with a database uniqueness constraint, not only with application logic. Two concurrent requests can pass an application check at the same moment; the database is the final arbiter. Third, record the outcome, not merely the fact that a key was seen. A request that failed definitively and a request that is still in progress are different states, and the retry behaviour should differ accordingly.

For card payments, Open Banking flows and account-to-account transfers, the same principle applies at different layers. The payment provider may have its own idempotency mechanism, but the internal order, invoice and ledger systems still need their own guardrails. A provider-side key protects the provider from duplicate charges. It does not protect your finance team from three order records pointing at one payment.

A webhook is a notification, not a receipt

Webhooks are frequently treated as authoritative events. They are better understood as prompts to go and check. A webhook may arrive twice, out of order or after a customer has already telephoned support. It may also arrive before the corresponding internal transaction has been committed.

A robust flow verifies the webhook signature, stores the raw event payload and processes it asynchronously. Deduplication should use the provider's event identifier where one exists, not a hash of the payload that may change between deliveries. Where the event signals a change in payment status, the system should fetch the authoritative status from the provider's API before updating the internal ledger. This costs an extra request, but it prevents a stale notification from overwriting a newer state.

Out-of-order delivery deserves particular attention. A payment can move from pending to authorised to captured, but the events may not arrive in that sequence. The internal state machine should reject backward transitions unless there is an explicit reversal or refund. If a "captured" event arrives before an "authorised" event, the system should not panic or create a duplicate. It should record what it knows and reconcile later.

Reconciliation is a product feature

In the UK, Faster Payments confirmations can be delayed, and customers can be quick to call when a payment does not appear immediately. Support teams under pressure may issue manual refunds to resolve a perceived double charge. Without a reliable reconciliation process, those manual actions become the source of the next problem.

Reconciliation should be a scheduled job with a clear owner, a defined tolerance and an escalation path. At its simplest, it compares the payment provider's settlement report with the internal ledger. It should identify payments that exist in one system but not the other, amounts that differ, and transactions that have remained pending beyond an agreed threshold. Each exception should be classified, not simply dumped into a spreadsheet.

The job should also be idempotent. Running it twice on the same day must not create duplicate adjustments. Corrections should be written as new ledger entries with a reason code, never as silent edits to historical records. This is how finance teams gain confidence: not by being told the system is correct, but by being able to see how exceptions were found and resolved.

Audit trails finance can defend

Application logs are written for engineers. Finance needs something different: an append-only record of commercial events, with correlation identifiers that link a customer action, a provider request, a webhook, a ledger entry and any manual intervention. Each entry should answer who or what acted, when, against which payment or invoice, and why.

The audit trail should be immutable from the product's perspective. If a correction is required, it is another entry. If a refund is issued, it references the original payment. If a support agent overrides a status, the reason is recorded. This may feel heavy for a small product, but payment disputes have a way of arriving months later, often when the original support agent has moved on.

A useful test is whether a finance manager can reconstruct a single customer's payment history without asking an engineer to query a database. If the answer is no, the audit trail is not yet doing its job.

Test disorder before customers find it

Duplicate and out-of-order events are difficult to reproduce manually, which is why they are often left untested. A payment test harness should be able to replay the same webhook twice, deliver events in reverse order, simulate a timeout after the provider has accepted the request, and insert an artificial delay between authorisation and capture. These scenarios belong in the automated test suite, not in a manual checklist completed the week before launch.

Contract tests against the provider's sandbox are useful, but they do not replace internal tests for your own state machine. The most valuable tests are the ones that assert the ledger remains balanced after a sequence of awkward events. If a duplicate webhook produces one order and one ledger entry, the system is behaving. If it produces two, the test has earned its place.

Design for the second attempt

The double-charge question is not really about payments. It is about whether the product can tolerate the real world. Messages will repeat. Confirmations will lag. Customers will refresh. Support will act before the system has caught up.

The measure of a payment flow is not whether a duplicate can occur. It will. The measure is whether the product can prove, quickly and without a spreadsheet, that the customer was charged once and recorded once. That is a defensible engineering position, and in 2026 it is the minimum a UK business should expect from software that handles its money.