Replacing a JS Hot-Path with Rust→WASM: 87× Faster Latency Histograms on CF Workers

We ship @stackbilt/llm-providers, a multi-provider LLM failover layer that runs natively in Cloudflare Workers. It tracks per-provider latency with a LatencyHistogram class used for observability polling and health checks.

Last week we replaced its pure-JS implementation with a thin adapter over @stackbilt/wasm-core, our new Rust→WASM package. The read path went from 1,426 ops/s to 123,641 ops/s — an 87× speedup — and dropped heap allocation from ~8 KB per call to near zero.

Here's what we learned.

The Problem

The original implementation was straightforward:

summary(provider: string): LatencySummary {
  const buffer = this.buffers.get(provider);
  const sorted = [...buffer].sort((a, b) => a - b); // 💸 copy + O(N log N) sort
  return {
    p50: sorted[Math.max(0, Math.ceil(0.50 * count) - 1)],
    p95: sorted[Math.max(0, Math.ceil(0.95 * count) - 1)],
    // ...
  };
}

Every summary() call on a provider with 1,000 samples:

1. Allocates a 1,000-element array (heap pressure)
2. Sorts it — O(N log N), ~10,000 comparisons
3. Reads three index positions and throws the sorted array away

Eviction used Array.shift() — O(N) — which shifts all 1,000 elements left on every write past the buffer limit.

With 6 providers and an observability poller calling allSummaries() on each health check, this compounds: 6 × (8 KB allocation + 10,000 comparisons) per poll.

The Fix: Rust→WASM Adapter

We built @stackbilt/wasm-core — a Rust crate compiled to WASM with wasm-pack --target bundler. The Rust LatencyHistogram uses:

• A sorted Vec<f64> with binary-search insertion — O(log N) insert, O(1) percentile read by index
• A circular index for eviction — O(1), no element shifting

The Rust percentile math is exact nearest-rank (ceil(p/100 * count) - 1), preserving the same semantics as the original.

The TypeScript side is a thin adapter — 50 lines including the LatencySummary interface:

import { LatencyHistogram as WasmLatencyHistogram } from '@stackbilt/wasm-core';

export class LatencyHistogram {
  private inner: WasmLatencyHistogram;
  private providers: Set<string> = new Set(); // needed for allSummaries()

  constructor(maxSamples: number = 1000) {
    this.inner = new WasmLatencyHistogram(maxSamples);
  }

  record(provider: string, latencyMs: number): void {
    this.providers.add(provider);
    this.inner.record(provider, latencyMs);
  }

  summary(provider: string): LatencySummary {
    return this.inner.summary(provider) as LatencySummary;
  }

  allSummaries(): Record<string, LatencySummary> {
    const result: Record<string, LatencySummary> = {};
    for (const provider of this.providers)
      result[provider] = this.inner.summary(provider) as LatencySummary;
    return result;
  }
  // ...
}

One thing to note: the WASM binding's all_summaries() method returned an empty object (a gap in the serde-wasm-bindgen serialization of HashMap). We worked around it by tracking the provider set on the JS side and calling summary() per provider — same result, clean semantics.

Results

Benchmarked with vitest bench, 6 providers × 1,000 samples:

Heap allocation: 3.3 KB across 100,000 summary() calls vs. ~8 KB per call on the JS path. The WASM path allocates only the small result object (~33 bytes); the sorted buffer lives in WASM linear memory.

The record() regression

WASM record() is 2.5× slower than JS. This is expected: V8 is extraordinarily optimized for Array.push() on Smi arrays, and each WASM call pays an FFI crossing. The binary-search insert is O(log N) vs. O(N) shift, but at 1,000 elements V8's shift is so fast the crossing overhead dominates.

This is an acceptable tradeoff. record() is called once per LLM response (write-rarely). summary() and allSummaries() are called on every health check and dashboard poll (read-often). Optimizing the read path is exactly right.

WASM in Vitest

Getting WASM to load in Vitest's Node.js environment required one config change — passing --experimental-wasm-modules to the fork pool:

// vitest.config.ts
pool: 'forks',
poolOptions: {
  forks: {
    execArgv: ['--experimental-wasm-modules'],
  },
},

All 487 existing tests passed unchanged. The WASM percentile math is exact, so no test expectations needed updating.

What's Next

This was Phase 0 — a proof of concept for the Rust→WASM pattern on CF Workers using @stackbilt/wasm-core. The same package is the target for Phase 1: spreading activation graph traversal in our AEGIS reasoning layer, where the compute profile is similar (pure CPU, no I/O, called on hot paths).

@stackbilt/llm-providers is open source. The WASM adapter landed in PR #101.