Design Exercise: Pastebin

Time to put theory into practice.

In this exercise, you will design a Pastebin-like service—a simple application that lets users store and share text snippets. This is a classic system design problem because it seems simple but reveals important design decisions as you dig deeper.

How to use this exercise:

1. Try designing the system yourself first (20-30 minutes)
2. Then read through the solution to compare approaches
3. Note where your thinking differed—both approaches might be valid

This mirrors the system design interview experience. There is no single "correct" answer, but there are better and worse approaches.

---

The Problem

Design a service like Pastebin where users can:

• Paste text and get a unique URL
• Access the paste via that URL
• Optionally set an expiration time

Seems simple, right? Let us see what complexity emerges.

---

Step 1: Clarify Requirements

Before drawing any boxes, understand what you are building.

Functional Requirements

Core features (must have):

• Users can create a paste with text content
• System generates a unique, short URL for each paste
• Anyone with the URL can view the paste
• Pastes can have an expiration time (optional)

Nice to have (ask interviewer):

• User accounts and authentication
• Private pastes (password protected)
• Edit/delete functionality
• Syntax highlighting
• Analytics (view count)

Out of scope (for this exercise):

• File uploads (images, documents)
• Real-time collaboration
• Version history

Non-Functional Requirements

Scale:

• How many pastes per day? → Let us assume 1 million pastes created per day
• How many reads per paste? → Average 10 reads per paste → 10 million reads/day
• How long do we store pastes? → 5 years for non-expiring pastes

Performance:

• Read latency: < 200ms
• Write latency: < 500ms (user can wait a bit)

Availability:

• 99.9% uptime (about 8 hours downtime per year)

Other:

• URLs should be short (for sharing)
• URLs should not be easily guessable (prevent enumeration)

---

Step 2: Back-of-the-Envelope Calculations

Let us size this system.

Storage Estimation

Pastes created:

plaintextCopy
1 million pastes/day × 365 days × 5 years = 1.8 billion pastes
Round to: 2 billion pastes

Size per paste:

plaintextCopy
Average paste size: 10 KB (text content)
Metadata (URL, timestamp, expiry, user): 100 bytes
Total per paste: ~10 KB

Total storage:

plaintextCopy
2 billion pastes × 10 KB = 20 TB
With 3x replication: 60 TB

60 TB is significant but manageable. This is not petabyte-scale.

QPS Estimation

Writes (paste creation):

plaintextCopy
1 million/day ÷ 100,000 seconds/day = 10 writes/second
Peak (3x): 30 writes/second

Reads (paste views):

plaintextCopy
10 million/day ÷ 100,000 seconds/day = 100 reads/second
Peak (3x): 300 reads/second

These numbers are modest. A single server could handle this, but we will design for growth and reliability.

URL Space

How many unique URLs do we need?

plaintextCopy
2 billion pastes over 5 years
Using base62 (a-z, A-Z, 0-9): 62 characters

6 characters: 62^6 = 56 billion combinations
7 characters: 62^7 = 3.5 trillion combinations

6-character URLs give us 56 billion combinations for 2 billion pastes—plenty of headroom. We will use 6 characters.

---

Step 3: High-Level Design

Start with the simplest architecture that could work:

plaintextCopy
┌─────────┐       ┌─────────────┐       ┌─────────────┐
│  User   │──────►│  API Server │──────►│  Database   │
└─────────┘       └─────────────┘       └─────────────┘

But we identified several requirements that complicate this:

• High availability (need redundancy)
• Fast reads (need caching)
• Large content storage (database might not be optimal)

Let us evolve the design:

plaintextCopy
                         ┌─────────────┐
                         │    Users    │
                         └──────┬──────┘
                                │
                         ┌──────▼──────┐
                         │     CDN     │ ← Cached pastes
                         └──────┬──────┘
                                │
                         ┌──────▼──────┐
                         │Load Balancer│
                         └──────┬──────┘
                                │
              ┌─────────────────┼─────────────────┐
              │                 │                 │
         ┌────▼────┐       ┌────▼────┐       ┌────▼────┐
         │  API 1  │       │  API 2  │       │  API 3  │
         └────┬────┘       └────┬────┘       └────┬────┘
              │                 │                 │
              └─────────────────┼─────────────────┘
                                │
              ┌─────────────────┼─────────────────┐
              │                 │                 │
         ┌────▼────┐       ┌────▼────┐       ┌────▼────┐
         │  Cache  │       │Metadata │       │ Object  │
         │ (Redis) │       │   DB    │       │ Storage │
         └─────────┘       └─────────┘       │  (S3)   │
                                             └─────────┘

Component Breakdown

Component	Purpose
CDN	Cache popular pastes at edge locations
Load Balancer	Distribute traffic across API servers
API Servers	Handle requests, generate URLs, coordinate storage
Cache (Redis)	Store hot pastes for fast reads
Metadata DB	Store paste metadata (URL, expiry, created_at)
Object Storage (S3)	Store paste content (the actual text)

Why separate metadata from content?

Paste content can be up to 10 KB or more. Storing large blobs in a relational database is inefficient. Object storage (S3, GCS) is designed for this:

• Cheap storage for large files
• Built-in replication and durability
• Easy CDN integration

Metadata is small and needs fast lookups—perfect for a database.

---

Step 4: Deep Dive - Key Components

URL Generation

We need unique, short, non-guessable URLs. Several approaches:

Option 1: Random Generation

plaintextCopy
Generate random 6-character string
Check if it exists in database
If exists, generate another
If not, use it

Pros: Simple, truly random (non-guessable) Cons: Collision checking adds latency and database load

Option 2: Counter + Base62 Encoding

plaintextCopy
Increment counter: 1, 2, 3, ...
Convert to base62: 1→"1", 62→"10", 3844→"100"

Pros: No collisions, fast Cons: Predictable (can enumerate), requires distributed counter

Option 3: UUID Truncation

plaintextCopy
Generate UUID: 550e8400-e29b-41d4-a716-446655440000
Take first 6 characters: 550e84
Base62 encode if needed

Pros: Stateless, no coordination needed Cons: Collision risk (mitigated by checking)

Recommended: Random with retry

For Pastebin's scale (10 writes/second), collision checking is cheap:

pythonCopy
def generate_url():
    while True:
        url = random_base62_string(6)
        if not database.exists(url):
            return url

With 56 billion possible URLs and 2 billion pastes, collision probability is ~3.5%. A single retry almost certainly succeeds.

Data Model

Paste Metadata (SQL Database):

sqlCopy
CREATE TABLE pastes (
    short_url VARCHAR(10) PRIMARY KEY,
    content_key VARCHAR(255),        -- S3 object key
    created_at TIMESTAMP,
    expires_at TIMESTAMP NULL,
    user_id VARCHAR(50) NULL,        -- Optional, for registered users
    view_count INT DEFAULT 0
);

CREATE INDEX idx_expires_at ON pastes(expires_at);

Paste Content (Object Storage):

plaintextCopy
Bucket: pastebin-content
Key: {short_url}
Content: Raw text content

API Design

Create Paste:

plaintextCopy
POST /api/pastes
Body: {
    "content": "print('Hello World')",
    "expires_in": 3600  // Optional: seconds until expiration
}

Response: {
    "url": "https://paste.example.com/abc123",
    "expires_at": "2024-12-20T10:00:00Z"
}

Read Paste:

plaintextCopy
GET /api/pastes/{short_url}

Response: {
    "content": "print('Hello World')",
    "created_at": "2024-12-19T10:00:00Z",
    "expires_at": "2024-12-20T10:00:00Z"
}

Or redirect to a rendered view:

plaintextCopy
GET /{short_url}
→ HTML page with paste content displayed

Read Path (Optimized)

For reads, we want to minimize latency:

plaintextCopy
1. User requests paste.example.com/abc123
2. CDN checks cache → HIT? Return immediately
3. If CDN MISS → Request goes to Load Balancer
4. API Server checks Redis cache → HIT? Return
5. If Redis MISS → 
   a. Fetch metadata from database
   b. Fetch content from S3
   c. Populate Redis cache
   d. Return response

With aggressive caching, most requests never hit the database.

Cache TTL Strategy:

• CDN: 5 minutes (balance freshness vs cache efficiency)
• Redis: 1 hour (hot pastes stay in memory)

Write Path

plaintextCopy
1. User submits paste content
2. API Server:
   a. Generate unique short URL
   b. Upload content to S3 (content_key = short_url)
   c. Insert metadata into database
   d. Return short URL to user
3. (Optional) Invalidate any cached versions

Write consistency: We insert metadata only after S3 upload succeeds. If S3 upload fails, we do not create a broken reference.

Expiration Handling

How do we delete expired pastes?

Option 1: Lazy deletion

plaintextCopy
When paste is requested:
    if paste.expires_at < now:
        return 404 (or "Paste expired")

Pros: Simple, no background jobs Cons: Expired data still consumes storage

Option 2: Background cleanup job

plaintextCopy
Every hour:
    DELETE FROM pastes WHERE expires_at < NOW()
    Delete corresponding S3 objects

Pros: Reclaims storage Cons: Need to manage background workers

Recommended: Both

• Check expiration on read (immediate user experience)
• Run background cleanup (reclaim storage)

---

Step 5: Addressing Bottlenecks

Database as Bottleneck

At 300 reads/second peak, a single database is fine. But if we scale to 10x:

Solution 1: Read replicas

plaintextCopy
Writes → Primary
Reads → Replicas (round-robin)

Solution 2: Cache more aggressively With 95% cache hit rate, only 5% of reads hit database:

plaintextCopy
3,000 reads/second × 5% = 150 database reads/second

S3 as Bottleneck

S3 is designed for massive scale. It is unlikely to be a bottleneck. But we can optimize:

• Use S3 Transfer Acceleration for global writes
• Use CloudFront (CDN) for reads
• Set appropriate cache headers

Hot Pastes

What if one paste goes viral and gets 1 million views?

The thundering herd problem:

• Paste is not in cache
• 10,000 concurrent requests all hit database/S3
• Database overwhelmed

Solution: Request coalescing

plaintextCopy
If paste not in cache:
    If another request is already fetching:
        Wait for that request to finish
    Else:
        Fetch and populate cache
Return cached value

Redis supports this with SETNX (set if not exists) for locking.

---

Step 6: Additional Considerations

Security

URL enumeration: With 6-character URLs, an attacker could try to guess valid URLs. Mitigations:

• Rate limiting per IP
• Use 7-8 character URLs (more combinations)
• Monitor for scanning patterns

Content moderation: People will paste malicious content. Consider:

• Abuse reporting mechanism
• Automated scanning for malware/illegal content
• Clear terms of service

Analytics

If we want view counts:

• Increment counter on each read
• Use Redis INCR (atomic, fast)
• Batch persist to database periodically

plaintextCopy
Read request:
    redis.incr("views:abc123")  // Fast, in-memory
    
Background job (every minute):
    for url, count in redis.scan("views:*"):
        database.increment_views(url, count)
        redis.delete("views:" + url)

Global Distribution

For a global user base:

• Deploy API servers in multiple regions
• Use global load balancer (AWS Global Accelerator, Cloudflare)
• S3 with cross-region replication
• Database: either multi-region (complex) or accept higher latency for writes

For Pastebin's read-heavy workload, CDN solves most latency issues.

---

Final Architecture Summary

plaintextCopy
                              ┌───────────────────────┐
                              │       Internet        │
                              └───────────┬───────────┘
                                          │
                              ┌───────────▼───────────┐
                              │         CDN           │
                              │   (CloudFront/CF)     │
                              └───────────┬───────────┘
                                          │
                              ┌───────────▼───────────┐
                              │    Load Balancer      │
                              └───────────┬───────────┘
                                          │
                    ┌─────────────────────┼─────────────────────┐
                    │                     │                     │
              ┌─────▼─────┐         ┌─────▼─────┐         ┌─────▼─────┐
              │  API Srv  │         │  API Srv  │         │  API Srv  │
              └─────┬─────┘         └─────┬─────┘         └─────┬─────┘
                    │                     │                     │
                    └─────────────────────┼─────────────────────┘
                                          │
            ┌─────────────────────────────┼─────────────────────────────┐
            │                             │                             │
      ┌─────▼─────┐                 ┌─────▼─────┐                 ┌─────▼─────┐
      │   Redis   │                 │  Primary  │                 │    S3     │
      │   Cache   │                 │    DB     │                 │  Storage  │
      └───────────┘                 └─────┬─────┘                 └───────────┘
                                          │
                                    ┌─────▼─────┐
                                    │  Replica  │
                                    │    DB     │
                                    └───────────┘

Cost Estimation (Rough)

Component	Monthly Cost (AWS)
3 API servers (t3.medium)	$100
RDS PostgreSQL (db.t3.medium)	$100
Redis (cache.t3.micro)	$25
S3 (60 TB storage)	$1,400
CloudFront (data transfer)	$500
Total	~$2,100/month

Not cheap, but reasonable for a service handling 10 million requests/day.

---

Common Mistakes in This Design

Mistake 1: Storing content in the database Large text blobs in PostgreSQL works but is inefficient. Object storage is cheaper and scales better.

Mistake 2: Forgetting about expiration Without expiration handling, storage grows forever. Always design for data lifecycle.

Mistake 3: Ignoring hot content One viral paste can overwhelm your system. Caching and request coalescing are essential.

Mistake 4: Over-engineering For 10 writes/second, you do not need Kafka, microservices, or sharding. Start simple.

---

Key Takeaways

1. Separate metadata from content. Different storage systems for different access patterns.

2. Cache aggressively. Read-heavy systems benefit enormously from CDN and in-memory caching.

3. Plan for expiration. Data without a lifecycle will consume resources forever.

4. Size your system. Back-of-envelope calculations prevent over-engineering and under-engineering.

5. Consider failure modes. Hot content, expired data, and storage failures all need handling.

---

Practice Extensions

Try extending this design:

1. Private pastes: How would you add password protection?
2. Edit functionality: What changes to support updating paste content?
3. Syntax highlighting: Where would this processing happen?
4. 10x scale: What breaks at 100 million pastes/day?

Work through these mentally or on paper before the next lesson.

---