Reliability Patterns for a Website Talking to Someone Else's System

The question isn't whether it will fail

Every website that exchanges data with an external system — a payment gateway, a government API, a CRM — will eventually get back something it didn't expect: a timeout, a malformed response, a connection that drops halfway through, or nothing at all. That can't really be engineered away; networks and third-party systems fail on their own schedule, not yours. The design question that actually matters is what happens next, and whether "what happens next" was a decision anyone made, or just whatever the default behaviour of an HTTP library happens to be.

Timeouts: deciding how long is too long

Left to its defaults, an HTTP client can wait far longer than anyone intended for a response that's never coming — tying up a server thread, a browser tab, or a customer's patience in the meantime. A timeout needs two separate values, not one: how long to wait for the connection to open, and how long to wait for a response once it has. A slow, overloaded API and a completely unreachable one fail differently, and a system that treats every failure identically — retrying a request that will never succeed, or giving up instantly on one that just needed another second — is guessing rather than deciding.

Retries with backoff, not just retries

Retrying a failed request is not, by itself, a strategy — it's an assumption that the failure was momentary, and that assumption is often wrong at exactly the moment it matters most: when the external system is struggling under load. If every client retries immediately on failure, the retries themselves add load to a system that's already failing, in a spike that arrives right when it's least able to absorb one. AWS's own architecture writing on the subject, much of it built on testing different retry strategies under contention, recommends exponential backoff with randomised jitter added to each wait interval, rather than a fixed delay: spacing retries out, and spreading them randomly rather than in lockstep, measurably reduces the total wasted work compared with retrying immediately or on a fixed delay (AWS Architecture Blog — Exponential Backoff and Jitter). The other half of the strategy is a hard cap: a maximum number of attempts and a maximum total wait, so a struggling dependency degrades a request rather than hanging it indefinitely.

Idempotency: why the same request can arrive twice

This is the piece most integrations get wrong, usually because the failure mode isn't intuitive until it's explained concretely. A client sends a request to create an order. The server receives it, processes it, and creates the order — but the response never makes it back to the client, because the connection drops a moment too soon. From the client's point of view, this is indistinguishable from the request never having been received at all; it has no way to know the order was actually created. If it follows a retry policy and resends the same request, and the server has no way of recognising it as a repeat, a second order gets created. Nobody made a mistake; the system behaved exactly as built. The duplicate is a structural consequence of retrying over an unreliable network, not a bug in any one component.

HTTP's own specification draws a real distinction here. RFC 9110 defines GET, HEAD, PUT, DELETE, OPTIONS, and TRACE as idempotent methods — meaning that, by definition, sending the same request multiple times has the same effect as sending it once — while POST and PATCH are not idempotent by default (RFC 9110, §9.2.2). That's why it's broadly safe for a browser or a proxy to silently retry a failed GET, and not safe to silently retry a POST that creates something new each time it succeeds — the specification simply doesn't promise the second attempt will behave the same as the first.

The practical fix for POST-like operations is an idempotency key: the client generates a unique identifier for each logical operation — a UUID, typically — and sends it with the request. The server records the outcome against that key, and if the same key arrives again, it returns the stored result instead of repeating the operation. This pattern is described in an IETF Internet-Draft, "The Idempotency-Key HTTP Header Field" (datatracker.ietf.org), which — worth being precise about — remains a draft rather than a ratified standard as of when this was checked. Its formal status doesn't change the underlying mechanism, which is already how a wide range of production APIs handle exactly this problem, standard or not: dedupe on a key the client controls, not on hoping the network behaves.

// Illustrative only — load secrets from environment variables, never hardcode them
const idempotencyKey = crypto.randomUUID();

await fetch(process.env.PAYMENT_API_URL, {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.PAYMENT_API_KEY}`,
    "Idempotency-Key": idempotencyKey,
  },
  body: JSON.stringify(orderPayload),
});

// A retry after a timeout reuses the same idempotencyKey,
// so the server can recognise it as the same logical operation.

Queues and dead-letter handling

Not every failure should be retried immediately by the same request that triggered it. For work that doesn't need to complete within the customer's page load — sending a confirmation, syncing a record to another system, submitting a filing — a queue sits between "the work that needs to happen" and "the system doing it," so a downstream outage delays the work instead of losing it. The queue retries on its own schedule, separate from the request that created the job, and a message that keeps failing after a set number of attempts gets moved somewhere it can be examined rather than retried forever or silently dropped. Amazon SQS's dead-letter queue documentation describes this purpose plainly: isolating messages that repeatedly fail to process so they can be inspected and diagnosed, rather than left to retry indefinitely or vanish (AWS documentation — Using dead-letter queues in Amazon SQS). The exact implementation varies by platform, but the underlying pattern — a defined place for the failures nobody has looked at yet — is worth building deliberately rather than discovering by accident when a queue fills up unnoticed.

Why "it worked in testing" is not evidence

Staging environments are usually faster, more reliable, and more forgiving than production, which is precisely why they're not a good test of failure handling. Sandbox credentials for an external API rarely replicate its production rate limits, its real outage patterns, or its response time under genuine load. And developers testing their own integration naturally exercise the path that's supposed to work far more than the ones that are supposed to fail. A working demo proves the happy path works. It says nothing about what happens when a request times out at exactly the wrong moment, arrives twice, or gets a response the code wasn't written to handle. The only way to know is to deliberately simulate those conditions — a forced timeout, a dropped connection, a duplicate submission — before go-live, and to keep watching for them afterwards, because integration failures in practice tend to show up under real load or during the external system's own maintenance windows, not during a scripted test.

If this is already live and nobody's sure

For a website that already talks to a payment gateway, a bank, or a government system, and where nobody is fully confident what happens when that connection fails mid-transaction, JagaWeb's Essential System Review (RM1,500, reduced to RM999 until 16 September 2026, excluding SST) can look specifically at how failures are currently handled and report on it plainly. Building a new integration with these patterns designed in from the start is Fixed-Scope Project work, from RM30,000.