Rebuilding UltraBalancer in Rust: What the Benchmark Actually Proves

UltraBalancer is a load balancer I built in Rust around a small goal: keep the forwarding path fast without making the operational path mysterious.

An earlier version of this article described a C/C++ design and a one-million-request-per-second result. That was not an honest description of the repository that exists today. The implementation is Rust, the project headline is 500K+ requests per second, and every number needs its workload beside it.

This is the corrected version.

What exists now

The public project includes:

• a Tokio-based Rust runtime;
• round-robin, least-connections, IP-hash, random, and weighted selection;
• active health checking and automatic failover;
• /metrics and /health endpoints;
• an admin API for changing backends;
• installable packages for Linux, macOS, and Windows.

The source and current installation instructions live in the UltraBalancer repository. The repository is the source of truth when this article and the code disagree.

The request path

The useful mental model is deliberately short:

listener
  -> connection task
  -> backend snapshot
  -> selection algorithm
  -> upstream connection
  -> streamed response

Health checks and administrative changes update backend state outside the hot path. A request should read the smallest practical view of eligible backends rather than coordinate with every health-checking task.

That split matters more than clever syntax. Contention in a shared health map or connection pool can erase gains from the forwarding loop.

Why Rust and Tokio

Rust gives the data path memory safety without a garbage collector, while Tokio provides the asynchronous I/O runtime. Neither automatically makes a program fast. The design still has to avoid unnecessary allocation, long-held locks, unbounded queues, and accidental blocking work inside async tasks.

The main rules are plain:

1. Keep request-owned state local to its task.
2. Share compact, read-heavy backend state.
3. Put active health mutation off the request path.
4. Bound queues and timeouts.
5. Reuse upstream connections where the protocol permits it.

Tokio documents that its scheduler is cooperative: a task that does not yield can delay other tasks. That is why CPU-heavy or blocking work must not quietly enter the forwarding path. See the Tokio runtime documentation for the runtime model.

Backend selection

The five algorithms share one selection boundary:

round robin       stable distribution across healthy backends
least connections favor the backend with less in-flight work
IP hash           preserve affinity for a client key
random            cheap distribution without shared rotation state
weighted          encode unequal backend capacity

The algorithm receives an eligible backend snapshot. It does not own health checks, connection establishment, or metrics. That separation keeps policy testable and makes it harder for one new algorithm to change unrelated behavior.

Health and failover

A backend is not healthy merely because a TCP connection opened once. The operational model needs to account for:

• active check failures;
• connection failures from real traffic;
• timeouts and slow responses;
• a threshold before removing or restoring a backend;
• the possibility that every backend is unhealthy.

The last case deserves an explicit response. Repeatedly trying a dead upstream set only converts a clear outage into slower failures and more work.

The published benchmark

The repository currently reports 500K+ requests per second and publishes the following Apple M4 Pro row:

These are project-reported numbers from a synthetic test. They are not an independent benchmark and they do not describe every production workload.

The number changes when any of these change:

• request and response size;
• HTTP version and connection reuse;
• TLS termination;
• upstream latency;
• logging and metrics volume;
• network topology;
• CPU frequency, core count, and operating system;
• failure rate and health-check activity.

A reproducible performance claim should publish the load generator, exact command, topology, payload, connection model, duration, latency distribution, error count, and hardware. Throughput without those details is a direction, not a guarantee.

What the benchmark does prove

It shows that the current Rust data path can sustain high request volume in the published test shape while keeping tail latency low on that machine.

It does not prove:

• identical performance with TLS and real payloads;
• production capacity for an unrelated service;
• resilience during upstream failure;
• fairness across long-lived and short-lived requests;
• safe operation at the same rate on every supported platform.

Those need separate tests.

Operations are part of the product

The less glamorous pieces matter when a process is on call:

• /health gives an orchestrator a narrow liveness surface;
• /metrics exposes traffic and latency for monitoring;
• the admin API changes the backend set without restarting the process;
• packages make the same release installable instead of requiring every user to reconstruct a build environment.

A fast binary that cannot explain which upstream is failing is not production-ready. It is a benchmark artifact.

What I learned

The headline is easy to overvalue

It is tempting to optimize the largest number in the README. P99 latency, errors, connection churn, and failure recovery are usually more useful capacity signals.

Health state is request-path state

Even when health checks run elsewhere, their data is read for every selection. Its representation and update strategy belong in the performance design.

Packaging catches different bugs

Cross-platform releases expose assumptions about certificates, paths, signals, and service management that a local benchmark never sees.

Corrections should remain visible

The original article made claims the current implementation did not support. Quietly changing the title would hide the useful lesson: performance writing needs the same verification discipline as performance code.

Sources and reproducibility

• UltraBalancer source, feature list, packages, and published benchmark
• UltraBalancer documentation
• Tokio runtime documentation

If you evaluate UltraBalancer for a real workload, rerun the test with your protocol, payload, upstream behavior, and failure modes. The published number is a starting hypothesis, not a capacity plan.