Rate Limiting in Practice

APIs eventually receive more requests than they can handle, from a broken retry loop, a scraper, or simply high usage. Rate limiting controls how many requests a client can make in a given period, so one client cannot exhaust the resources other clients depend on. This post covers what rate limiting does, the common algorithms, a minimal implementation, and where to enforce it.

What rate limiting actually solves

Rate limiting caps how many requests a client can make in a given time window. It protects system capacity.

Without it:

• One client can exhaust a shared resource (database, third-party API, GPU), blocking requests from other clients

• Costs scale with the volume of abusive traffic, not legitimate usage, especially on metered upstream APIs (e.g. OpenAI, Anthropic)

• A client bug (an infinite retry, a bad polling loop) can cause an outage instead of a minor issue

Rate limiting is a basic API control, alongside authentication and authorization. Authentication identifies the caller, authorization decides what it can do, and rate limiting decides how much and how fast.

The main algorithms

There are four common algorithms.

Token bucket is the most common choice for APIs because because real-world traffic is naturally bursty. It gives legitimate users the flexibility of a quick burst (e.g., loading a page with multiple assets) while keeping a firm grip on the long-term average.

A minimal token bucket

Basically, a token bucket needs two numbers per client: how many tokens are left, and when it was last refilled. Everything else is arithmetic.

type Bucket = { tokens: number; lastRefill: number };
type Result = { allowed: boolean; retryAfter: number }; // retryAfter in seconds

// Note: In a real production app, use an LRU (Least Recently Used) cache with a TTL
// to prevent this Map from growing infinitely.
const buckets = new Map<string, Bucket>();

const CAPACITY = 20;        // Max burst size
const REFILL_RATE = 5;      // Tokens added per second

function allow(clientId: string): Result {
  const now = Date.now() / 1000; // Current time in seconds
  
  // 1. Get existing bucket, or initialize a new one with full capacity
  const bucket = buckets.get(clientId) ?? { tokens: CAPACITY, lastRefill: now };

  // 2. Refill tokens based on time elapsed since the last request
  const elapsed = Math.max(0, now - bucket.lastRefill);
  bucket.tokens = Math.min(CAPACITY, bucket.tokens + elapsed * REFILL_RATE);
  bucket.lastRefill = now;

  // 3. Determine if the request is allowed and consume a token if so
  const allowed = bucket.tokens >= 1;
  if (allowed) {
    bucket.tokens -= 1;
  }

  // 4. Save state (required for new clients. Existing ones update by reference)
  buckets.set(clientId, bucket);
  
  return {
    allowed,
    retryAfter: allowed ? 0 : Math.ceil((1 - bucket.tokens) / REFILL_RATE),
  };
}

When a request is denied, retryAfter tells the client how long to wait. 1 - bucket.tokens is how many tokens are missing to reach one, dividing by REFILL_RATE converts that into seconds, and Math.ceil rounds up to whole seconds, since the Retry-After header takes an integer.

This version keeps state in memory, which works for a single process. In production, that state has to live somewhere shared, like Redis. Because multiple API servers might handle requests for the same user simultaneously, Redis check needs to be atomic (executed as a single, uninterrupted operation) to prevent race conditions from bypassing the limits.

In practice, most teams don't hand-write this logic for production. Instead, they use proven tools:

• Node.js: Libraries like rate-limiter-flexible (Redis-backed for distributed setups) or framework middleware like express-rate-limit.

• .NET: ASP.NET Core's built-in Microsoft.AspNetCore.RateLimiting middleware (supports token bucket, sliding window, and partitioning out of the box). This article covers how to implement rate limiting in the framework.

• Infrastructure: Edge rate limits using NGINX's limit_req module or the CDN's built-in rules.

Where to enforce it

• Gateway / reverse proxy (nginx, API gateway, Cloudflare): Catches abuse before it reaches application code, saving compute. This is the right place for coarse, high-volume limits (e.g., max 100 req/sec per IP).

• Application layer: Needed when limits depend on business logic. This is where per-user plan tiers are enforced, per-endpoint costs, or dynamic limits based on request payload size (Often used alongside gateway limits for defense in depth).

• Client-side: never a substitute for server-side enforcement, but respecting Retry-After and spacing out retries stops clients from hitting limits unnecessarily.

A few common issues:

• Keying by IP alone breaks for users behind shared NAT or corporate proxies. Prefer an API key, token, or user ID when one exists.

• Better to return 429 Too Many Requests with a Retry-After header. A bare rejection forces clients to guess, often leading to aggressive retries.

• Rate limiting is not a substitute for authentication or input validation. It protects capacity, not correctness.

Takeaway

Rate limiting is a lightweight pattern that significantly improves reliability and controls cost. Use a token bucket at the gateway, key it by something more stable than IP, and return clear headers so clients know the limit and when to retry.

The main implementation challenge is sharing state correctly across multiple instances. That's why, unless one have unusual requirements, reach for an already written solution rather than writing one from scratch: a gateway plugin (Kong, Envoy, Cloudflare) for coarse limits, or a library like rate-limiter-flexible in Node.js for application-level control.

Thanks for reading! If you have any suggestions, corrections, or feedback, feel free to send an email to contact@curiousthoughts.blog.