An SFU looks simple in one sentence: receive RTP packets from publishers and forward selected streams to subscribers without decoding and recompositing them.

The difficult parts arrive around that sentence—signaling races, packet loss, backpressure, congestion, session ownership, recording, and observability.

This article is an architecture note from work on Surf Media Engine. An earlier version called it production-grade and published precise latency, memory, and capacity numbers without a reproducible test report. I removed those claims. The design is useful; the old table was not evidence.

Surf Media Engine components from signaling through media routing and bounded recording
The media path, control path, and recording path fail differently and should be bounded separately.

Why an SFU

For a room with multiple participants, the common topologies are:

  • Mesh: each participant sends media directly to every other participant. Client upload work grows with the room.
  • MCU: a server decodes, mixes, and re-encodes media. Clients receive a composed stream, but the server pays the codec cost and adds latency.
  • SFU: each participant uploads one stream or set of simulcast layers. The server forwards selected packets to subscribers without mixing them.

The SFU moves bandwidth and routing responsibility to the server while avoiding a full transcode for every forwarded track.

For N participants, a simple directed mesh has N(N-1) sender-to-receiver media relationships. A basic SFU shape has N uplinks and roughly N downlinks, though the server still performs fan-out work for every subscribed track.

Keep control and media separate

The control path handles:

  • rooms and participant identity;
  • SDP offers and answers;
  • ICE candidates;
  • track subscription decisions;
  • authorization and lifecycle.

The media path handles:

  • RTP packet ingress;
  • track lookup;
  • subscriber fan-out;
  • RTCP feedback;
  • queueing and congestion behavior.

A slow database call in the control path should not block packet forwarding. Likewise, a subscriber that cannot consume packets quickly enough must not grow an unbounded queue inside the media path.

A small routing core

The useful data structure is a map from a published track to the subscribers currently receiving it.

use bytes::Bytes;
use dashmap::DashMap;
use tokio::sync::broadcast;

pub struct Router {
    tracks: DashMap<String, broadcast::Sender<Bytes>>,
}

impl Router {
    pub fn publish(&self, track_id: &str, packet: Bytes) -> usize {
        let Some(sender) = self.tracks.get(track_id) else {
            return 0;
        };

        let receivers = sender.receiver_count();
        if receivers == 0 {
            return 0;
        }

        let _ = sender.send(packet);
        receivers
    }
}

Bytes clones share the underlying immutable buffer, so moving packet payloads between tasks can avoid copying the entire packet. Tokio's broadcast channel gives every active receiver each value, but it is bounded: a slow receiver can lag and lose old values.

That lag behavior is important. Real-time media usually prefers dropping stale packets to accumulating seconds of latency, but the policy should be explicit and measured.

The send lookup can be constant-time on average; forwarding the packet to N subscribers is not free or magically O(1). Network writes, per-subscriber encryption and protocol state, feedback, and congestion decisions still scale with the fan-out.

See the bytes::Bytes documentation and Tokio broadcast channel documentation for the exact clone and lag semantics.

Renegotiation is a state machine

Adding or removing tracks can require negotiation. The failure-prone version creates an offer whenever a track changes and assumes no other offer is in flight.

Two peers can create offers at the same time. This is known as glare. A robust design tracks whether it is making an offer, knows whether it should be polite or impolite during a collision, and uses rollback where supported.

The browser-side pattern looks roughly like this:

let makingOffer = false;

pc.onnegotiationneeded = async () => {
  try {
    makingOffer = true;
    await pc.setLocalDescription();
    signaling.send({ description: pc.localDescription });
  } finally {
    makingOffer = false;
  }
};

The complete pattern also handles incoming description collisions and ICE candidates. MDN's perfect negotiation guide is a better reference than inventing a partial state machine from memory.

On the SFU side, serialize changes to each peer connection. A room-wide lock is usually too broad; a per-participant negotiation queue makes the ownership clearer.

Jitter estimation is not a playout policy

RTP packets do not arrive at perfectly even intervals. RFC 3550 defines an interarrival jitter estimator:

J(i) = J(i-1) + (|D(i-1,i)| - J(i-1)) / 16

The 1/16 term smooths the observed transit-time variation. It does not say that a receiver should set its buffer to exactly two times J, nor does it promise a particular delivery probability.

An adaptive buffer still needs policy for:

  • minimum and maximum delay;
  • reordered and duplicate packets;
  • late-packet rejection;
  • when to request retransmission;
  • when to conceal or skip a missing packet;
  • how quickly to grow or shrink the target delay.
Packets arriving with variable delay, entering an ordered bounded buffer, and leaving on a steady playout schedule
The estimator describes network variation; the buffer policy decides how much latency to trade for continuity.

The RTP specification, RFC 3550 is the source for the jitter calculation.

NACK has a deadline

Generic NACK lets a receiver ask for missing RTP packets. Retransmission helps only when the replacement can arrive before its playback deadline.

A useful decision needs:

  • packet sequence and age;
  • estimated round-trip time;
  • current target delay;
  • whether the sender retains the packet;
  • the media type and concealment behavior.

For a packet already too old to play, a retransmission consumes bandwidth without improving the call. The feedback path must understand time, not merely gaps.

The feedback message format is standardized in RFC 4585.

Bound every subscriber

One slow or disconnected subscriber should not stall the publisher or the room.

For each subscriber, bound:

  • pending packet count or bytes;
  • time spent waiting for a transport write;
  • retransmission history;
  • control messages;
  • task lifetime after disconnect.

When the bound is exceeded, drop according to media policy, reduce subscribed layers, or disconnect the subscriber. Unbounded buffering converts packet loss into process memory growth and unusable latency.

Recording is a separate backpressure problem

Recording should not write synchronously from the live forwarding task. A safer pipeline is:

media event
  -> bounded recording queue
  -> muxer or segment writer
  -> bounded upload queue
  -> object storage
  -> durable manifest

Each arrow needs a failure decision.

  • If storage is slow, how many segments may wait locally?
  • If the queue is full, does recording stop or may it affect the live call?
  • How is a partial recording marked?
  • Can the final manifest be rebuilt after a worker restart?

The live call should remain the higher-priority path unless the product explicitly promises otherwise.

Shared state should not pretend to be media state

Redis or a database can store room ownership, participant metadata, and leases. It should not sit in the packet path.

For multi-instance operation, the hard questions are:

  • Which worker owns the room?
  • How does a signaling request reach that worker?
  • What happens when the owner disappears?
  • Is media state reconstructed, migrated, or dropped?
  • How long can a stale lease survive?

A key-value record does not solve failover by itself. The system needs explicit ownership, fencing, and recovery behavior.

Observability that helps during a bad call

Aggregate CPU and request counts are not enough. Useful media metrics include:

  • packets and bytes in and out by track type;
  • packet loss and NACK rate;
  • jitter and round-trip-time distributions;
  • subscriber queue depth and drops;
  • negotiation failures by signaling state;
  • active peer connections and tracks;
  • recording queue depth and failed segments;
  • room placement and worker saturation.

Keep labels bounded. Participant and track IDs belong in sampled traces or logs with a retention policy, not in unbounded Prometheus labels.

What I would benchmark now

Instead of publishing one capacity row, I would report a matrix:

audio-only rooms vs audio + video
participants and subscribed tracks per room
codec and bitrate
simulcast layers
packet loss, jitter, and RTT profiles
encrypted packet rate
CPU, memory, queue drops, and network throughput
join, leave, and renegotiation churn
recording enabled and disabled

The test should run long enough to expose leaks and queue growth, and it should publish the generator, topology, software revision, and error totals.

Lessons I kept

Signaling races deserve tests

Renegotiation succeeds in the happy-path demo and fails under simultaneous joins, quick mute/unmute cycles, and network reconnection. Test the state transitions, not only the first connection.

Backpressure is a product decision

Every bounded queue eventually fills. The code must decide what to drop and what experience the user sees.

"Zero-copy" needs a boundary

Sharing a Bytes buffer avoids one application-level copy. Encryption, kernel buffers, and network interfaces still do work. State exactly which copy disappeared.

Rust removes classes of bugs, not systems work

Ownership helps make task and buffer lifetimes explicit. It does not design congestion control, choose queue bounds, or verify the deployment.

Sources

An SFU is not one clever router. It is a collection of bounded state machines connected by deadlines. That is what makes it interesting—and what makes honest measurements more valuable than a large unsupported capacity claim.