API Gateway: The Single Front Door for Microservices

Split a monolith into 40 services and you inherit a new problem: every one of those services now needs TLS, token verification, rate limiting, logging, and CORS. Copy that code 40 times and you get 40 slightly-different implementations of the same security perimeter — one of which is broken. The API gateway exists to solve that duplication. It is the single ingress that every external request passes through, where the cross-cutting concerns live exactly once.

Think of it as the front door of the building rather than 40 doors on 40 offices. The door checks your ID, logs your entry, and tells you which floor to go to. What you do once you're inside is the office's business.

What an API gateway actually does#

A gateway is a reverse proxy with opinions. Raw traffic arrives at a public address, and before anything reaches a backend the gateway runs a policy pipeline:

• TLS termination — decrypt once at the edge so backends speak plaintext HTTP inside the trusted network.
• Authentication — verify the JWT signature, check expiry, or introspect an OAuth token, so no backend re-implements crypto.
• Rate limiting and quotas — cap requests per API key so one abusive client can't starve the rest.
• Routing and path rewriting — map POST /api/v2/orders to the orders service and strip the public prefix.
• Protocol translation — accept HTTP/1.1 from the browser and speak gRPC to the backend.
• Aggregation — fan one client request out to several services and merge the results.
• Observability — emit one consistent set of metrics, traces, and access logs for every request that enters the system.

The reason all of this belongs at the edge and not in each service is enforcement. A perimeter you implement in one place is a perimeter you can reason about; one you copy into every service is a perimeter with holes.

Isn't that just a load balancer?#

This is the first follow-up in almost every interview, and the distinction is real. A load balancer operates at L4 or L7 and distributes connections — it inspects an IP, a port, maybe a Host header, and picks a healthy backend. It does not understand that /v2/orders requires the orders:write scope, that this API key is over its hourly quota, or that this one client request should trigger three backend calls.

An API gateway is application-aware. It terminates the client protocol, applies per-route policy, and can call multiple backends for a single request. The two are not competitors — in most real deployments a load balancer sits in front of the gateway.

The clean mental model: the load balancer answers "which gateway instance takes this connection?", and the gateway answers "is this request allowed, and where does it go?"

Rate limiting: which algorithm and where do the counters live?#

Rate limiting is the deep-dive interviewers reach for, because a naive answer breaks under bursty traffic. The counters cannot live in one gateway instance's memory — you run many instances behind the load balancer, so the limit has to be shared. The standard answer is a low-latency store like Redis, keyed by <api_key, route, window>, with an atomic INCR plus EXPIRE on each request. That adds roughly a millisecond per request but makes the limit global.

The algorithm choice is a trade-off between accuracy, memory, and burst behavior:

Token bucket wins most arguments. It stores just two integers — the current token count and the last refill timestamp — refilled lazily on read, and it lets a client burst up to the bucket size while still enforcing an average rate over time. Reach for a leaky bucket only when the downstream needs a strictly uniform arrival rate regardless of how bursty the client is.

Where does authentication belong?#

The rule worth memorizing: authenticate at the edge, authorize where the data lives.

Authentication proves identity — verify the JWT signature, check that it hasn't expired, introspect the OAuth token against the auth server. That is pure crypto with no domain knowledge, so it happens once at the gateway. The gateway then strips the public token and forwards trusted internal context — signed headers like X-User-Id and X-Scopes, sent over mTLS so a backend can trust them.

Authorization is different. "Can this user edit this order?" needs to know who owns the order, which is domain knowledge that lives in the service. So fine-grained authorization stays in the backend. The gateway only enforces coarse, route-level checks — "this endpoint requires the orders:write scope" — because that decision needs nothing but the token.

Putting full authorization in the gateway forces it to understand every service's data model, which turns the edge into a bottleneck that every team has to modify. Keep it dumb about domains and smart about identity.

Aggregation and the Backend-for-Frontend pattern#

A mobile client on a slow connection does not want to make six round trips to render one screen. The gateway can expose a Backend-for-Frontend (BFF) endpoint that fans one request out to the profile, orders, and recommendations services in parallel, merges the responses, and returns a single payload shaped for that client.

The upside is fewer round trips and dumb clients. The cost is that the gateway now holds business logic, which makes it a deployment bottleneck — every client-shape change is a gateway change. Two failure modes need explicit handling: a slow backend can stall the whole aggregate response unless each fan-out call has its own timeout, and a failed backend should degrade to a partial response rather than a total error. GraphQL is essentially a declarative version of this same fan-out-and-merge pattern.

Keeping one bad backend from sinking the gateway#

Because every request flows through it, the gateway is a shared-fate component: one slow upstream can consume every worker thread and take down traffic to healthy services too. The defenses are layered:

• Bulkheads — give each upstream its own bounded connection and thread pool, so an exhausted payments pool can't starve search.
• Timeouts — per-route, and shorter than the client's timeout, so the gateway gives up before the caller does.
• Retries with a budget — jittered exponential backoff, capped so retries never exceed ~10% of traffic. Uncapped retries turn a blip into a self-inflicted flood.
• Circuit breakers — after the error rate crosses a threshold, open the breaker and fast-fail for a cooldown instead of piling requests onto a dying service.
• Load shedding — return 429 at the edge under queue pressure rather than accepting work you can't finish.

Gateway or service mesh?#

Once you pass ~50 services, the question becomes what stays at the gateway and what moves into a service mesh (sidecar proxies like Envoy on every pod). The split is directional: north-south traffic at the gateway, east-west traffic at the mesh.

The trap here is duplicating retries in both layers. If the gateway retries three times and the mesh retries three times, one failing call can become nine — the retry amplification problem. Pick one layer to own retries for a given hop.

How this shows up in interviews#

Interviewers rarely ask "what is an API gateway?" and stop there. They ask it, accept the list of responsibilities, and then push on one edge. Expect: "why not just a load balancer?", "which rate-limiting algorithm and where do the counters live?", "where does auth happen and what do you forward to the backend?", and — for senior loops — "you have 50 services, what moves to a mesh?"

The strongest answers name a concrete trade-off and the condition that would reverse it. Token bucket by default, leaky bucket when the downstream demands smooth flow. Authenticate at the edge, authorize in the service. Retries at the gateway or the mesh, never both. Walk one request end to end — TLS terminates, JWT verifies, rate limit checks Redis, route resolves, backend responds, access log emits — and you've shown you understand the gateway as an operable system, not a box in a diagram.

Further learning#

• Reverse Proxy vs API Gateway vs Load Balancer — ByteByteGo's video that draws the exact boundary this article walks through.
• API Gateway — ByteByteGo's written deep dive on gateway responsibilities and patterns.
• Related roadmap topics: Scaling, Database Indexing, SQL vs NoSQL, and CAP Theorem.

You can also practice this topic on YeetCode to work through the gateway decisions interactively.

FAQ#

What is the difference between an API gateway and a reverse proxy?#

Every API gateway is a reverse proxy, but not every reverse proxy is a gateway. A plain reverse proxy (NGINX in its simplest config) forwards requests to a backend and maybe adds TLS or caching. A gateway adds application-level policy on top: per-route authentication, per-API-key rate limiting, request aggregation, protocol translation, and version routing. The gateway understands your API's contract; a bare reverse proxy just moves bytes.

Does adding an API gateway create a single point of failure?#

It concentrates traffic, but it isn't a single point of failure if you run it correctly. You deploy multiple stateless gateway instances behind a load balancer, so losing one instance just reshuffles connections. The shared state that matters — rate-limit counters — lives in a replicated store like Redis, not in gateway memory. The real risk is shared fate from a slow backend, which is why bulkheads, per-route timeouts, and circuit breakers are non-negotiable: they stop one failing upstream from consuming the whole gateway's capacity.

How do you roll out a breaking API change without breaking existing clients?#

Version the contract — either in the path (/v2/orders) or via an Accept header — and run v1 and v2 as separate routes behind the same gateway. Often the gateway rewrites old v1 requests to the v2 backend through a shim so you maintain only one implementation. Announce the timeline with Deprecation and Sunset response headers, emit per-API-key usage metrics so you can see who is still on v1, and dark-launch v2 to a small traffic percentage using gateway canary rules. Public APIs typically keep the old version alive 6–12 months before retiring it.

Should authentication and authorization both happen at the gateway?#

Authentication should; full authorization should not. Verifying a token's signature and expiry is pure cryptography with no knowledge of your domain, so doing it once at the edge saves every backend from re-implementing it. Authorization — deciding whether this specific user may act on this specific resource — needs to know who owns the resource, which is domain data that lives in the service. The gateway enforces only coarse, scope-level checks; the service makes the fine-grained ownership decision.

Why use a token bucket instead of a fixed window for rate limiting?#

A fixed window resets its counter at a hard boundary, so a client can send a full window's worth of requests just before the reset and another full window right after — effectively 2x the limit in a short span. A token bucket avoids that edge burst by refilling tokens continuously at the average rate while still allowing a controlled burst up to the bucket size. It costs only two integers per key (token count and last-refill time), refilled lazily, which makes it both cheaper and smoother than tracking a full sliding-window log.