URL Shortener (bit.ly)

Every QR code on a cereal box, every tweet with a link, every SMS campaign from your bank — they all run through a URL shortener. bit.ly processes ~10 billion clicks per month. The system looks trivial: take a long URL, return a 7-character string. But at scale it becomes a masterclass in read-heavy systems, cache warming, hash collision handling, and hot-partition avoidance. Interviewers love it because the surface area is small but the depth is unlimited.

1. POST /shorten — given a long URL, return a unique short code (e.g. https://sho.rt/aB3xKz)
2. GET /{code} — redirect the user to the original long URL (HTTP 301 or 302)
3. Custom aliases — user can request a vanity slug (e.g. /my-brand)
4. Link expiry — optional TTL after which the short URL returns 410 Gone
5. Analytics — track click count, country, referrer, device (eventual consistency is fine)

• Full analytics dashboards (we store events; aggregation is a separate service)
• User authentication / link ownership (assume API-key auth at gateway)
• Link previews / safety scanning (separate async pipeline)

Assumptions:

• 500M redirects/day, 5M new URLs/day
• Average long URL: 200 bytes; short URL record: ~500 bytes
• 80% of traffic hits 20% of links (Pareto)

Key insight: This is overwhelmingly read-heavy. The cache hit ratio is the single most important performance lever.

                    ┌──────────────────────────────────────┐
                    │              Clients                  │
                    └────────────┬─────────────────────────┘
                                 │ HTTPS
                    ┌────────────▼─────────────────────────┐
                    │          API Gateway                  │
                    │   (rate-limit, auth, TLS termination) │
                    └──────┬──────────────┬────────────────┘
                           │              │
               ┌───────────▼──┐    ┌──────▼───────────┐
               │  Write Path  │    │   Read Path       │
               │  (Shortener) │    │  (Redirect Svc)   │
               └───────┬──────┘    └──────┬────────────┘
                       │                  │  cache miss
               ┌───────▼──────┐    ┌──────▼────────────┐
               │  ID Generator│    │   Redis Cluster   │
               │  (Snowflake) │    │   (L1 cache)      │
               └───────┬──────┘    └──────┬────────────┘
                       │                  │ still miss
               ┌───────▼──────────────────▼────────────┐
               │          Primary DB (Cassandra)        │
               │   Partition key: short_code            │
               └───────────────────────────────────────┘
                                 │ async
               ┌─────────────────▼──────────────────────┐
               │         Analytics Event Bus (Kafka)     │
               └────────────────────────────────────────┘

Write path: Client → API Gateway → Shortener Service → ID Generator (Snowflake) → Base62 encode → write to Cassandra + warm Redis

Read path: Client → API Gateway → Redirect Service → Redis lookup (hit: 302 immediately) → on miss: Cassandra → populate Redis → 302

The most common mistake is MD5/SHA256-then-truncate. That gives you a fixed output for a given URL — which means two users shortening the same URL get the same code (good for dedup, bad if you want user-scoped expiry). The cleaner approach is ID-based encoding:

1. Generate a globally unique 64-bit integer via Snowflake (timestamp + datacenter + worker + sequence)
2. Base62-encode it ([0-9][a-z][A-Z] = 62 chars)
3. A 64-bit integer needs at most ceil(log62(2^64)) = 11 chars; we cap at 7 for aesthetics (62^7 = 3.5 trillion codes)

This is collision-free by construction — Snowflake guarantees uniqueness. No DB round-trip to check.

// Java 17 — Base62 encoder using records
public record ShortCode(String value) {
    private static final String ALPHABET =
        "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
    private static final int BASE = ALPHABET.length(); // 62

    public static ShortCode fromId(long id) {
        if (id <= 0) throw new IllegalArgumentException("id must be positive");
        var sb = new StringBuilder();
        while (id > 0) {
            sb.append(ALPHABET.charAt((int)(id % BASE)));
            id /= BASE;
        }
        return new ShortCode(sb.reverse().toString());
    }

    public long toId() {
        long id = 0;
        for (char c : value.toCharArray()) {
            id = id * BASE + ALPHABET.indexOf(c);
        }
        return id;
    }
}

301 vs 302: Use 302 Found (temporary redirect). 301 is cached permanently by browsers — you lose all analytics after the first visit. 302 hits your servers every time; that’s what you want.

API contract:

GET /{code}
→ 302 Location: https://original-long-url.com/path
→ 404 if code unknown
→ 410 Gone if expired

Store in the same table with a is_custom flag. On write: check existence first (Cassandra lightweight transaction — INSERT IF NOT EXISTS). Charge custom aliases against a user’s quota at the gateway.

Cassandra table (optimised for point lookups by short_code):

Secondary access pattern — “list all links for user_id”: separate table links_by_user (user_id, created_at, short_code) — materialised view in Cassandra or a separate write on creation.

Indexes: None on the primary table — Cassandra discourages secondary indexes at this scale. Use the materialised view pattern.

Partitioning: short_code as partition key distributes evenly. Avoid using user_id as partition key on the main table — a power user would create a hot partition.

TTL: Set Cassandra’s native TTL on write for expiring links. No background sweep job needed.

Conclusion: Snowflake + Base62. Decentralised, monotonically increasing (good for time-range queries), zero collision risk.

Conclusion: 302 for analytics-first products. 301 only if you’re shutting down and want graceful offload.

Conclusion: Cassandra for writes + Redis for reads. Accept eventual consistency on analytics.

This system prioritises AP (Availability + Partition Tolerance). A short redirect returning a stale URL is far less harmful than a 503 error. We tolerate eventual consistency on click counts; redirect correctness is protected by Redis + Cassandra replication.

AuthN/AuthZ: API key per client at the gateway. Write endpoints require a valid key. Read (redirect) is public — no auth needed.

Encryption: TLS 1.3 in transit. Cassandra encryption at rest (AES-256). Redis TLS + AUTH.

Input validation:

• URL must parse as valid http:// or https:// — reject javascript:, data:, ftp: schemes
• Max URL length: 2,048 chars (standard browser limit)
• Custom alias: alphanumeric + hyphens only, 3–30 chars, reserved words blocklist (api, admin, health)

Abuse / Phishing: Async URL scanner (Google Safe Browsing API) on write. Flag malicious URLs; short code still created but returns a warning interstitial (like bit.ly’s spam page).

Rate limiting: 100 writes/min per API key (token bucket at gateway). 10,000 reads/min per IP (sliding window in Redis).

PII/GDPR: Long URLs may contain PII (e.g. search queries). Log only short_code in access logs, never the long URL. Analytics events store hashed IP. Right-to-erasure: delete row + Redis DEL + tombstone in Cassandra.

Audit log: All write operations (create, delete, update expiry) written to immutable audit Kafka topic.

• Unique short URLs created per hour (anomaly = abuse spike)
• Click-through rate per link (dashboard)
• Top 100 hottest links by click volume (real-time Flink job)

OpenTelemetry spans on every redirect: gateway → redirect-svc → redis-lookup → cassandra-fallback. Trace ID injected into analytics event for correlation. Sample at 1% baseline + 100% on errors (tail-based sampling via Grafana Tempo).

• Single Redirect Service + single Redis + single Postgres
• Snowflake with one worker node
• What breaks first: Postgres read throughput at ~2K RPS

• Migrate to Cassandra (3-node cluster, RF=3)
• Redis Cluster (3 shards)
• Horizontal scale Redirect Service (stateless, 5+ pods)
• What breaks first: Redis hot keys for viral URLs

• Add local in-process cache (Caffeine, 1s TTL) in each Redirect pod — absorbs viral URL spikes without Redis
• CDN layer (Cloudflare) for top 1M hot codes — serve redirects from edge, zero origin hits
• Cassandra expand to 9 nodes across 3 AZs
• What breaks first: CDN cache invalidation on link deletion/expiry

• Full CDN-first: 95%+ of redirects served at edge via Cloudflare Workers (KV store)
• Cassandra multi-region active-active (2 regions, LOCAL_QUORUM)
• Snowflake workers per datacenter (datacenter ID in worker config)
• Analytics moves to Flink + Iceberg (real-time + historical)
• What breaks: cross-region consistency for link deletion — use tombstone TTL + CDN purge API

Brownfield integration: Enterprises often need to keep their own domain (links.acme.com). Support custom domains via CNAME → your CDN. Each domain maps to a tenant namespace in Cassandra.

Build vs Buy:

• ID generation: Build Snowflake (trivial, no vendor dependency)
• Cache: Buy — Redis Enterprise or Elasticache; operational complexity not worth owning
• Analytics pipeline: Buy — Confluent (Kafka managed) + Databricks or BigQuery
• CDN: Buy — Cloudflare or Fastly; edge redirect latency you can’t match in-house

Multi-tenancy: Partition by tenant_id as the first token in Cassandra (composite partition key: (tenant_id, short_code)). Each tenant gets rate limits, custom domain, and analytics namespace.

Vendor lock-in: Cassandra is open-source — no lock-in. If using DynamoDB, abstract behind a UrlRepository interface so you can swap. Redis is replaceable with Dragonfly or KeyDB if licensing becomes a concern.

TCO ballpark (1B redirects/month):

• 3× Cassandra i3en.2xlarge = ~$1,500/mo
• Redis Cluster 3× r6g.xlarge = ~$900/mo
• Redirect Service pods (10× t3.medium) = ~$300/mo
• Kafka (Confluent serverless) = ~$200/mo
• CDN (Cloudflare Pro) = ~$200/mo
• Total: ~$3,100/mo — $0.003 per 1,000 redirects

Conway’s Law: Split ownership naturally: Platform team owns ID generation + Cassandra. Product team owns Redirect + Shortener services. Data team owns analytics pipeline. Avoid the monolith temptation on Day 1 — the redirect path is hot enough to justify its own deployment.

1. Clarify before drawing: Ask “Is dedup important? Should two users shortening the same URL get the same code?” — it completely changes your hash vs Snowflake decision.

2. Lead with the read/write ratio: Saying “this is 100:1 read-heavy, so the entire design is a cache in front of a DB” immediately signals senior-level thinking.

3. Justify 301 vs 302 explicitly — most candidates miss this. It’s a 10-second point but interviewers notice.

4. Avoid the MD5 trap: Truncated hashes have collision probability that grows faster than people expect. If asked, say “I’d use ID-based encoding to guarantee uniqueness without a DB round-trip.”

5. Name the thundering herd scenario: “A viral tweet sends 500K clicks in 60 seconds to one short code — here’s how I protect Redis and Cassandra.” Then explain singleflight / local cache / CDN. This separates candidates who’ve operated systems from those who’ve only designed them on paper.

Fluency vocabulary: Base62, Snowflake ID, singleflight, write-around cache, read-through cache, Cassandra TTL, LOCAL_QUORUM, 302 vs 301, CDN cache invalidation, hot partition, thundering herd.

• Cassandra Data Modeling — Official guide; partition key selection is directly applicable here
• Bitly Engineering: Building a Reliable URL Shortener — Real-world lessons from the team that runs billions of redirects
• Snowflake ID — Twitter Engineering — Original announcement and design rationale
• RFC 7231 §6.4.3 — Formal definition of 302 Found and caching semantics