system-design · beginner

TCP vs UDP — Reliable Streams vs Lightweight Datagrams

The Central Question

Consider two real-time network applications operating over the internet:

Application A is an online banking transfer service. If a single byte representing the money amount is dropped or received out of order, financial data becomes corrupt.
Application B is a multiplayer first-person shooter game transmitting player movement coordinates 60 times per second. If a movement packet sent 100 milliseconds ago is lost, retransmitting that stale coordinate is useless — the player has already moved to a new position.

Using the same transport protocol for both applications would lead to catastrophic failures.

At Layer 4 of the OSI model, two transport protocols dominate network communication: TCP (Transmission Control Protocol) and UDP (User Datagram Protocol).

This lesson answers one central question: How do TCP (connection-oriented, reliable in-order byte stream) and UDP (connectionless, low-overhead datagrams) govern transport-layer data delivery, and how do engineers choose between TCP reliability guarantees and UDP low-latency execution?

Defining the Transport Layer: Ports and Demultiplexing

While IP (Internet Protocol) routes packets between computer IP addresses, transport layer protocols (TCP and UDP) utilize Port Numbers (e.g. Port 80, 443, 53) to deliver data to specific software processes running on a host.

flowchart TD
  IPPacket[Network IP Packet<br/>Src IP: 203.0.113.5 | Dst IP: 198.51.100.10] --> Demux{Transport Layer Header}
  Demux -->|TCP Port 443| WebServer[Web Server Process - Nginx]
  Demux -->|TCP Port 5432| DBServer[PostgreSQL DB Process]
  Demux -->|UDP Port 53| DNSServer[CoreDNS Process]

Figure 1: Transport layer port demultiplexing routing packets to application processes.

Ephemeral Ports and Network Address Translation (NAT)

When a client application initiates an outgoing socket connection to a server, the operating system kernel dynamically assigns a temporary **Ephemeral Port** (typically in the range 49152–65535) to identify the local client socket.

When traffic passes through home routers or cloud NAT gateways, Network Address Translation (NAT) maps multiple internal private IP addresses and ephemeral ports to a single public IP address. NAT routers maintain state tables tracking active TCP connection states and UDP datagram mappings, allowing response packets from public servers to be routed back accurately to the correct internal client device.

TCP: The Reliable Connection-Oriented Byte Stream

TCP presents application code with a continuous, ordered, reliable stream of bytes. TCP guarantees that data sent across a network arrives completely, un-duplicated, and in the exact sequence it was transmitted.

flowchart LR
  subgraph TCP Core Guarantees
    G1[1. Connection Setup: 3-Way Handshake]
    G2[2. Reliable Delivery: ACKs & Automatic Retransmission]
    G3[3. In-Order Sequencing: Sequence Numbers]
    G4[4. Flow & Congestion Control: Sliding Windows]
  end

Figure 2: The four core reliability mechanisms provided by TCP.

1. TCP 3-Way Handshake Connection Setup

Before transmitting application bytes, TCP establishes a virtual connection between client and server via a **3-Way Handshake**:

sequenceDiagram
    autonumber
    actor Client as Client Socket
    participant Server as Server Socket
    
    Note over Client,Server: Phase 1: TCP 3-Way Handshake Setup
    Client->>Server: SYN (Seq = x)
    Server-->>Client: SYN-ACK (Seq = y, Ack = x + 1)
    Client->>Server: ACK (Seq = x + 1, Ack = y + 1)
    Note over Client,Server: Connection ESTABLISHED (Data can flow)
    
    Note over Client,Server: Phase 2: TCP 4-Way Teardown (Termination)
    Client->>Server: FIN (Seq = x + 100)
    Server-->>Client: ACK (Ack = x + 101)
    Server->>Client: FIN (Seq = y + 200)
    Client-->>Server: ACK (Ack = y + 201)
    Note over Client,Server: Connection CLOSED

Figure 3: Sequence diagram detailing the TCP 3-way handshake setup and 4-way teardown.

2. Sequence Numbers, Acknowledgments (ACKs), and Retransmission

TCP breaks application byte streams into segments, assigning an incremental **Sequence Number** to every byte.

The receiver sends Acknowledgment (ACK) packets confirming received byte ranges.
If the sender does not receive an ACK before a Retransmission Timeout (RTO) fires, it automatically retransmits the missing segment.

3. Flow Control and Congestion Control

Flow Control (Sliding Window): Prevents a fast sender from overwhelming a slow receiver's memory buffer.
Congestion Control (Cubic / BBR): Prevents senders from flooding network routers, slowing down transmission when packet loss or bufferbloat is detected.

Modern Congestion Control: TCP BBR vs. Loss-Based Cubic

Traditional TCP congestion control algorithms (like TCP Cubic) treat packet loss as the sole signal of network congestion. When a single router buffer overflows and drops a packet, Cubic slashes the sender's transmission window by 50%.

In modern cloud networks with high-speed links and deep router buffers, loss-based algorithms suffer from severe latency degradation known as Bufferbloat.

Modern infrastructure platforms deploy BBR (Bottleneck Bandwidth and RTT):

flowchart TD
  BBR[Google TCP BBR Algorithm] --> Model1[1. Measures Bottleneck Bandwidth: Max delivery rate]
  BBR --> Model2[2. Measures Min Round-Trip Time: Physical propagation delay]
  
  Model1 --> MaxThroughput[Paces transmissions at exact link capacity WITHOUT filling router queues]
  Model2 --> MaxThroughput

Figure 4: BBR congestion control model pacing traffic to prevent bufferbloat.

Why BBR Outperforms Cubic

Instead of filling router queues until packets drop, BBR continuously models the physical bottleneck bandwidth and minimum RTT of the network path. It paces packet transmissions to maximize throughput while keeping queue lengths near zero, reducing p99 tail latency by up to 10x over lossy networks.

UDP: Lightweight, Connectionless Datagrams

UDP is a minimal, connectionless transport protocol. It wraps application data in a lightweight header containing source/destination ports and a checksum, then transmits the packet directly onto the network as an independent Datagram.

flowchart TB
  App[Application Data Payload] --> UDPHeader[UDP Header: Ports + Length + Checksum]
  UDPHeader --> IP[IP Packet Output]
  
  style UDPHeader fill:#fff3cd,stroke:#ffebaa

Figure 5: Anatomy of a lightweight UDP datagram.

What UDP Does NOT Provide

Zero Handshake: UDP requires no connection setup. Data is transmitted immediately (0-RTT).
No Delivery Guarantees: If a UDP packet drops in transit, UDP does not retransmit it.
No Ordering Guarantees: Datagrams may arrive out of sequence or duplicated.
No Congestion Control: Senders transmit at full rate regardless of network congestion.

Head-of-Line (HOL) Blocking: The Latency Penalty of TCP

Because TCP guarantees strict in-order byte delivery, a single dropped packet stalls all subsequent data bytes arriving behind it:

sequenceDiagram
    autonumber
    actor Client as Receiver Socket Buffer
    participant Net as Network Link
    participant Server as Sender
    
    Server->>Net: Send Packet 1, Packet 2, Packet 3
    Net-->>Client: Packet 1 Arrives (Delivered to App)
    Net--xClient: PACKET 2 DROPPED IN TRANSIT!
    Net-->>Client: Packet 3 Arrives (Held in TCP Buffer)
    
    Note over Client: Head-of-Line Blocking!<br/>Packet 3 cannot be delivered to application<br/>until Packet 2 is retransmitted & ACKed!
    
    Server->>Net: Retransmit Packet 2
    Net-->>Client: Packet 2 Arrives
    Note over Client: Buffer Unlocked! Packet 2 & 3 delivered to App.

Figure 6: Sequence diagram illustrating Head-of-Line (HOL) blocking in TCP.

Architectural Comparison: TCP vs. UDP

Feature / Vector	TCP (Transmission Control Protocol)	UDP (User Datagram Protocol)
Connection Model	Connection-Oriented (Requires Handshake).	Connectionless (No Setup).
Data Format	Continuous Byte Stream.	Discrete Datagram Messages.
Delivery Guarantee	Guaranteed (Automatic Retransmissions).	Best-Effort (Un-guaranteed).
Ordering Guarantee	Guaranteed (In-order Sequence Numbers).	Un-ordered (Packets may arrive out of order).
Head-of-Line Blocking	Yes (Packet loss stalls subsequent bytes).	No (Datagrams processed independently).
Header Size	20 Bytes (up to 60 with options).	8 Bytes.
Congestion Control	Built-in (Cubic, BBR).	None (Application responsibility).

Socket API Pseudocode: Code-Level Differences

Application code interacts with TCP and UDP using distinct OS socket types:

1. TCP Socket Pseudocode (`SOCK_STREAM`)

import socket
Create TCP Stream Socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(('0.0.0.0', 8080))
server_socket.listen(128) # Listen for incoming TCP handshakes
Accept connection (Completes 3-Way Handshake)
client_socket, client_address = server_socket.accept()
Read from continuous byte stream
data = client_socket.recv(4096)
client_socket.sendall(b"HTTP/1.1 200 OK\r\n\r\n")
client_socket.close()

2. UDP Socket Pseudocode (`SOCK_DGRAM`)

import socket
Create UDP Datagram Socket
udp_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp_socket.bind(('0.0.0.0', 5353))
Receive discrete datagram payload directly (No connection setup)
data, client_address = udp_socket.recvfrom(512)
Send discrete datagram payload back
udp_socket.sendto(b"\x81\x80ResponseData", client_address)

Complete Worked Example: CheckoutLab Platform Protocol Selection

Let's inspect the transport protocol selection across different services in the CheckoutLab platform (checkoutlab.com).

flowchart TD
  Platform[CheckoutLab Network Platform] --> Service1[1. REST Payment API]
  Platform --> Service2[2. Live Telemetry Stream]
  Platform --> Service3[3. Internal DNS Resolution]
  
  Service1 -->|Requires 100% Data Integrity| TCP1[TCP + TLS 1.3 - Port 443]
  Service2 -->|Requires Lowest Latency - Drop Old Stats| UDP1[UDP Metrics Collector - Port 8125]
  Service3 -->|Requires Fast 1-RTT Small Lookups| UDP2[UDP DNS - Port 53]

Figure 7: Decision tree selecting transport protocols for diverse platform workloads.

Failure Modes and Engineering Mitigations

Failure Scenario	Root Cause	System Symptom	Detection Metric	Mitigation Strategy
1. TCP Head-of-Line Stalling	Unstable cellular network causes 5% packet loss on an HTTP/2 connection.	Web app freezes; all parallel multiplexed HTTP streams stall waiting for TCP retransmission.	High TCP retransmit rates and elevated p99 user response latency.	Upgrade transport to HTTP/3 (QUIC over UDP) to isolate packet loss to individual streams.
2. Un-controlled UDP Network Flood	Custom UDP application sends data at 10 Gbps without congestion control logic.	Network routers drop packets globally, causing severe bufferbloat for adjacent services.	High packet drop rates on network switches and router buffer overflows.	Implement application-level congestion control and rate limiting on custom UDP protocols.
3. TCP Idle Connection Drop	Stateful firewall or NAT router drops idle TCP socket state after 5 minutes.	Client app attempts to reuse socket, resulting in connection reset errors (`RST`).	High `connection_reset_by_peer` error spikes on long-lived connections.	Implement TCP Keep-Alive probes (`SO_KEEPALIVE`) or application-level ping/pong heartbeats.
4. Assuming TCP Implies Application Idempotency	Developers assume TCP reliability guarantees that business transactions execute once.	Network socket drops after server processes mutation; client retry double-executes charge.	Duplicate transaction records in application database.	Enforce application-level `Idempotency-Key` headers on all state-mutating HTTP POST requests.

What You Should Remember

TCP is a reliable in-order byte stream: TCP uses handshakes, ACKs, retransmissions, and flow control to guarantee complete, ordered data delivery.
UDP is a lightweight datagram protocol: UDP sends independent messages without handshakes, ACKs, or retries, offering minimal latency overhead.
TCP causes Head-of-Line (HOL) blocking: Dropped TCP packets stall all subsequent bytes in the stream until the missing segment is retransmitted.
BBR prevents bufferbloat: Modern BBR congestion control paces TCP transmissions based on bottleneck bandwidth rather than packet loss.
TCP reliability does not equal application idempotency: TCP guarantees byte delivery; application code must still enforce idempotency keys to handle retries safely.

Glossary of Terms

Term	Definition
TCP (Transmission Control Protocol)	A connection-oriented transport protocol providing reliable, ordered byte stream delivery.
UDP (User Datagram Protocol)	A connectionless transport protocol providing lightweight, best-effort datagram delivery.
Port Number	A 16-bit integer in transport headers used to route data packets to specific application processes.
Ephemeral Port	A temporary high-numbered port assigned dynamically by the operating system kernel for outgoing client connections.
3-Way Handshake	The TCP connection setup sequence consisting of `SYN`, `SYN-ACK`, and `ACK` packets.
Head-of-Line (HOL) Blocking	The delay phenomenon where a lost packet stalls the delivery of subsequent arrived packets in a stream.
TCP BBR	A congestion control algorithm that paces transmissions based on bottleneck bandwidth and round-trip time.

Practice Scenario and Self-Assessment

Architecture Scenario

You are designing the networking architecture for an online multiplayer battle-royale game. The game engine requires:

Player Authentication & Store Purchases (Must be 100% reliable)
Real-time Player Position Synchronization (60 updates/sec, low latency)
In-Game Text Chat (Reliable text delivery)

**Questions**:

Recommend TCP or UDP for each of the three game networking requirements and justify your choice.
Explain how Head-of-Line blocking would degrade gameplay if player position synchronization were implemented over standard TCP connections.

Interactive Self-Assessment

UDP avoids TCP Head-of-Line blocking delays, allowing applications to drop stale packets rather than freezing for retransmissions.

UDP automatically encrypts all voice data using hardware AES encryption.

UDP requires a complex 5-way handshake that verifies audio quality.

UDP guarantees that audio packets always arrive in perfect sequential order.

Subsequent arrived packets are held in the OS buffer and cannot be delivered to application code until the missing packet is retransmitted.

The OS buffer immediately deletes all subsequent arrived packets.

The TCP socket connection is forcibly closed with an HTTP 500 error.

The TCP socket automatically converts the stream into UDP datagrams.

What to Learn Next

HTTP and HTTPS — Protocols, Headers, and Security: Learn how HTTP semantics operate over TCP and TLS streams.
DNS — Domain Name System Resolution and Caching: Explore how domain names resolve to IP addresses before transport connections open.
Load Balancing — Algorithms and Layers: Master Layer 4 TCP vs Layer 7 HTTP load balancing.

Track: Reliability and Operations

Previous: Single Point of Failure (SPOF) — Identifying and Eliminating SPOFs

Next: Thundering Herd

By Shubham Jain

All articles · Study paths

Shubham Jain · Learning Lab