API Design for High‑Throughput Systems: Rate Limiting, Versioning, Idempotency

Featured image

Building an API that survives real traffic is more than writing fast code. When a service that handled 100 RPS suddenly sees 10 000 RPS, the failure modes are often predictable: aggressive client retries, traffic spikes, downstream back‑pressure, and duplicate side‑effects. The three patterns that prevent those failures—rate limiting, versioning, and idempotency—are not optional extras; they are table stakes for any production‑grade API.

---

1. Rate Limiting – Protecting Your System From Yourself

Why it matters

Rate limiting is frequently described as a defense against malicious bots, but the far more common scenario is self‑inflicted overload. A flash sale, a push notification that drives millions of users to the same endpoint, or a buggy third‑party integration that retries in a tight loop can all saturate a service that was never designed for sustained high volume.

Core algorithms

Algorithm	How it works	When to use
Token bucket	Each client gets a bucket that refills at a fixed rate; each request consumes a token. Buckets have a maximum capacity, allowing short bursts.	External APIs where occasional spikes are acceptable but sustained traffic must be limited.
Leaky bucket	Requests are queued and released at a constant output rate, regardless of arrival rate.	Situations that need a steady downstream throughput, e.g., protecting a write‑heavy database.
Fixed window	Counts requests in a discrete time window (e.g., 1 000 per minute) and resets at the boundary.	Simple internal services where edge‑case burst‑over‑window traffic is not a concern.
Sliding window	Similar to fixed window but uses a moving time slice, eliminating the “burst at the edge” problem.	Most public APIs; higher implementation cost but more accurate enforcement.

Where to enforce it

Place the limiter as early as possible—typically at the API gateway (Kong, AWS API Gateway, Nginx rate‑limit module). Doing it at the gateway prevents the request from consuming any application resources. If you must enforce it in‑app, use a lightweight middleware that checks a distributed store (Redis, DynamoDB) before any business logic runs.

Response contract

• Status: 429 Too Many Requests
• Headers:
  • Retry-After: <seconds|HTTP‑date> – tells the client when to try again.
  • X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset – give well‑behaved clients visibility into their quota.

Designing a clear contract lets good clients back off intelligently, reducing retry storms that amplify the original overload.

---

2. API Versioning – Evolving Without Breaking

The problem

Once a client (mobile app, partner integration, third‑party developer) depends on an endpoint, any breaking change risks a cascade of failures. Versioning isolates those risks, but the strategy you pick influences maintainability and developer experience.

Common strategies

Strategy	URL shape	Pros	Cons
URI versioning	/v1/orders	Immediate visibility, easy routing at the gateway.	Can lead to parallel codebases (v1, v2) that drift apart.
Header versioning	Accept: application/vnd.myapi.v2+json	Keeps URLs clean; semantics belong in the representation.	Less discoverable, requires extra routing logic, harder to test in a browser.
Query‑parameter versioning	/orders?version=2	Quick to implement, easy to test.	Mixes versioning with resource addressing; not suitable for public APIs.

When to bump a version

• Additive change (new optional field) – no version bump.
• Removal, rename, or type change – increment version.
• Semantic change (same field name, different meaning) – treat as breaking and version.

Deprecation workflow

1. Release the new version (/v2).
2. Add Deprecation and Sunset response headers to /v1 requests.
3. Log and monitor usage of the old version.
4. Communicate directly with remaining clients.
5. After the agreed window (commonly 6 months for public APIs, 3 months for internal), retire the old version.

Treat versioning as a communication problem as much as a technical one; clear deprecation timelines and headers are essential.

---

3. Idempotency – Making Retries Safe

Why retries happen

Network timeouts, load‑balancer failovers, and mobile connectivity loss all cause clients to retry a request they think may have failed. For read‑only GETs this is harmless, but for writes it can cause duplicate side‑effects (double charges, duplicate orders, etc.).

The idempotency key pattern

1. Client generates a UUID (or other globally unique value) and sends it in a header, e.g., Idempotency-Key: 123e4567‑e89b‑12d3‑a456‑426614174000.
2. Server checks a fast key‑value store (Redis, DynamoDB) for that key.
   • If the key exists, return the stored response (status code + body).
   • If not, process the request, store the key → response mapping with a TTL, then return the response.
3. TTL choice – 24 hours for payment‑type APIs, up to 7 days for long‑running async workflows.

What to store

• Idempotency key
• HTTP status code
• Full response body (or a reference to it)
• Optionally the request payload for validation – if a later request uses the same key but a different payload, return 422 Unprocessable Entity.

Multi‑step operations

When an operation spans several downstream calls (charge a card, update inventory, send email), the idempotency guarantee must cover the entire saga. A practical approach:

• Make each downstream call itself idempotent (e.g., use upserts, conditional writes).
• Persist a saga state record in your primary database, not just a cache, so that a retry can resume from the last successful step.
• Return the final aggregated response once the saga completes; subsequent retries read the saga state and return the same result.

Client‑generated vs server‑generated keys

Client‑generated keys are essential because the client is the party that experiences the failure and decides to retry. Server‑generated keys would require the client to have already received a successful response, defeating the purpose.

---

4. How the Three Patterns Interact

• Rate limiting shapes the load that reaches your service.
• Idempotency guarantees that the reduced load can be safely retried without corrupting state.
• Versioning lets you evolve both the limiter and the idempotency implementation without breaking existing consumers.

Real‑world scenario

Imagine a B2B payments API used by dozens of Indonesian accounting platforms:

• Rate limits are applied per API key (not per IP) because each client may issue thousands of requests on behalf of many end users.
• Idempotency is mandatory on all POST/PATCH endpoints; keys are stored in Redis with a 24‑hour TTL.
• Versioning follows a URI scheme (/v1/payments, /v2/payments). A six‑month deprecation window gives partners time to migrate.

The decisions reinforce each other: the per‑key limit protects the payment gateway, idempotency prevents double charges during retries, and versioning lets you tighten limits or change the idempotency storage strategy without breaking existing integrations.

---

5. Documentation – The Often‑Forgotten Glue

Even a perfectly engineered API will fail in production if developers cannot discover the contract. Remember:

• OpenAPI defines the schema but does not explain why a 429 might be returned or when an Idempotency-Key is required.
• Include usage examples, error‑handling guides, and retry‑backoff recommendations.
• Publish the rate‑limit values and deprecation timelines in a human‑readable format (markdown site, developer portal).

---

6. Build vs. Buy – Where to Spend Engineering Effort

Concern	Buy (gateway, SaaS)	Build (in‑app)
Rate limiting	API Gateway (Kong, AWS, Apigee) – cheap, battle‑tested, handles IP/API‑key granularity.	Custom middleware – high cost, often duplicate effort.
Version routing	Most gateways support path‑based routing; header routing is also available.	Possible but adds complexity.
Idempotency	No generic service; must be implemented in your domain logic.	Required – store keys, handle saga state, enforce payload consistency.

The rule of thumb: buy everything you can, build what is domain‑specific. For most teams the only custom piece is the idempotency layer.

---

7. FAQ

Q: What status code should I return when a request is rate limited? A: 429 Too Many Requests with a Retry-After header. Without the header, well‑behaved clients cannot back off intelligently, leading to retry storms.

Q: How do I make a multi‑step operation idempotent? A: Treat the whole flow as a saga. Persist a saga state record, make each downstream call idempotent, and store the final response keyed by the client‑provided idempotency key.

Q: Should internal services be versioned? A: Use explicit versioning only when services are deployed independently. Otherwise, contract testing (Pact, consumer‑driven contracts) or shared schema libraries (protobuf, Avro) are lighter weight.

Q: What granularity should rate limits have? A: Combine approaches: unauthenticated endpoints → per‑IP; authenticated user‑facing endpoints → per‑user; B2B APIs → per‑API‑key, with tiered limits per endpoint.

Q: How long should I keep idempotency keys? A: Long enough to cover the longest realistic retry window. 24 hours is common for payments; 7 days for long‑running async workflows. Adjust based on storage pressure.

---

8. Closing Thought

Rate limiting, versioning, and idempotency are rarely the headline features of a new product, but they are the difference between an API that scales gracefully and one that triggers 2 am fire‑drills. The patterns are well understood, the tooling is mature, and the engineering cost is modest compared to the cost of emergency incident response and customer refunds.

Build them in from day one, document them clearly, and let your future on‑call self thank you.

---

For a deeper dive into idempotency, see Stripe’s guide on the topic: https://stripe.com/docs/api/idempotent_requests.