Every Binance Spot pair on one machine: a Rust L1 engine

The brief was narrow on purpose. Take the best bid and ask of every spot pair on Binance, keep them in memory as integers, and on every change re-evaluate only the triangles that pass through the pair that changed. Emit an event when a cycle clears fees. Place no orders. And from the first run, measure what this costs on one machine: events per second, microseconds per stage, staleness against the exchange clock, drops, reconnects. The measuring was the point. A bot that cannot say how late its own view of the market is cannot be reasoned about, and this one was going to be reasoned about hard.

This post is the engine. The next three are what it found when pointed at the market: where a book update spends its first two milliseconds, why the taker loses, and a maker study.

The shape of the problem

On 2026-10-12 Binance Spot had 1,375 trading pairs over 519 assets. Quote currencies are lopsided: 506 pairs quote in USDT, 310 in TRY, 256 in USDC, 38 in BTC, 12 in ETH. Those pairs form 1,728 triangles, asset triples with all three pairs listed, and each triangle can be traded in two directions, so with a reasonable set of start assets the engine watches 3,230 cycles.

The number that shapes the design is fan-out: how many triangles contain a given pair. The mean is 4.3. The maximum, for USDT/TRY, is 309 triangles, 618 cycles. A thin TRY pair that updates a few times a minute costs almost nothing to re-evaluate; USDT/TRY updates hundreds of times a second and every update touches 618 cycles. If both live on the same thread, every USDT/TRY tick delays the thin legs by the hub's whole evaluation, and the thin legs are where the signal is.

So the connection plan separates them. Any pair with fan-out above a threshold (50 by default) goes on a dedicated hub connection; in practice that is one connection with four symbols. The rest are sorted by 24-hour trade count and dealt snake-wise, 0, 1, …, N−1, N−1, …, 0, across enough connections to stay under a per-connection stream budget, so busy pairs spread evenly. Each connection gets its own OS thread, pinned to a core when I give it one. An ownership table maps every symbol to exactly one thread, and only that thread ever writes the symbol's slot.

Rules for the hot path

Four rules, all checkable:

• No heap allocation after startup. Connection, handshake and subscribe may allocate; once the first frame arrives, nothing does.
• No floating point. Prices and quantities are mantissas at the symbol's own tick and step exponent, and the evaluation is u128 arithmetic.
• No async runtime. The crate has no tokio. One blocking thread per connection reads a socket; the service side runs on ordinary threads.
• No locks except a seqlock per symbol, single writer, any number of readers.

The reason is not religion. Each of these removes a source of tail latency that would otherwise show up in the per-stage histograms and be indistinguishable from the exchange's own jitter. I wanted every tail I saw to be the exchange's, and with these rules it was.

The frame

The transport is deliberately small: a TcpStream with TCP_NODELAY, rustls with webpki-roots, an HTTP upgrade written by hand and parsed with httparse, and my own RFC 6455 reader. The reader owns one 256 KiB buffer, grows it only if a frame ever exceeds it (counted when it happens; it never has), reassembles fragmented messages, and hands back a frame that borrows from the buffer:

pub struct Frame {
    pub opcode: Opcode,
    pub len: usize,
    payload: Payload, // range into the reader's buffer, or the fragment buffer
}
 
impl<S: Read + Write> WsReader<S> {
    pub fn next_frame(&mut self) -> io::Result<Frame> { /* … */ }
    pub fn payload(&self, f: &Frame) -> &[u8] { /* … */ }
}

The writer side, pong, subscribe and close, allocates freely; it runs a few times per session. Compression is never requested, and a server that offers it anyway fails the handshake loudly rather than silently costing a decompression per frame.

The loop around it is the whole hot path. Read a frame, stamp t0, and dispatch on the opcode: binary frames are SBE market data, text frames are subscribe acks, errors, or the serverShutdown notice Binance sends before a planned disconnect; pings get their payload echoed back; a close or a silence longer than the watchdog ends the session and the thread reconnects with jittered backoff. Binance pings every 20 seconds and expects a pong within 60, allows 5 client messages per second per connection, and closes connections after 24 hours, so subscriptions go out 200 streams per message, 250 ms apart, and connection i of N plans its own reconnect at 23.5 h × (i+1) / N so they never go dark together.

Decoding SBE

Binance publishes market data as SBE, Simple Binary Encoding, over the same WebSocket. A BestBidAskStreamEvent is an 8-byte header followed by a fixed block and a length-prefixed symbol:

The decoder is a pure function from a byte slice to an update, with every surprise counted rather than thrown: unknown template, unknown schema, a symbol the registry has not seen, trailing bytes after the symbol, and a precision mismatch when the exponent on the wire differs from the symbol's tick size and the mantissa has to be rescaled. Those counters are the first thing I read after any live run; they all have to be zero, and they have been.

let symbol = reg.lookup(&body[sym_start..sym_start + sym_len]).ok_or(DecodeError::UnknownSymbol)?;
let info = reg.symbol(symbol);
let px_exp = body[16] as i8;
let qty_exp = body[17] as i8;
let (bid_px, m1) = normalize(i64_at(body, 18), px_exp, info.tick_exp).ok_or(DecodeError::OutOfRange)?;
let (bid_qty, m2) = normalize(i64_at(body, 26), qty_exp, info.step_exp).ok_or(DecodeError::OutOfRange)?;

Symbol lookup is a hash of a fixed 16-byte key, the uppercase name zero-padded; the longest symbol on the exchange is 15 bytes, and startup fails if that ever changes. Building the key from the frame is a stack copy. Normalizing to the symbol's own exponents at decode time means every later computation knows its bounds: the largest price mantissa on the exchange is 10¹¹, so a product of three fits comfortably in u128.

There is a JSON fallback for running without an API key. It is a byte scanner, not a serde deserializer: it finds the six keys it needs in the envelope and parses decimal strings digit by digit into a mantissa at the target exponent. It is not slower than SBE in practice, which surprised me, and it is the reason the first weeks of measurement needed no credentials.

The slot

Each symbol's latest quote lives in one cache line:

#[repr(C)]
pub struct L1 {
    pub bid_px: i64, pub bid_qty: i64, pub ask_px: i64, pub ask_qty: i64,
    pub event_time_us: i64, // 0 when the feed has none
    pub update_id: i64,
    pub recv_ns: u64,       // monotonic, when the frame was received
}
 
#[repr(C, align(64))]
pub struct Slot {
    seq: AtomicU32,
    data: UnsafeCell<L1>,
}

Fifty-six bytes of data, a sequence counter, 64-byte alignment. The writer bumps seq to odd, writes the data with a volatile store, bumps it to even, with release fences between. A reader loads seq, spins if it is odd, copies the data, fences, reloads seq, and retries if it moved. There is one writer per slot by construction and readers never dereference, only copy, so a torn copy is possible and is simply discarded. The unit test for this runs three reader threads against 200,000 writes of a value whose seven fields must all be equal, and asserts they never see a mismatch. It is the only concurrency primitive on the hot path, and it is the reason a cycle spanning three connections can be evaluated by any of their threads without a lock.

The decision

Evaluation is two phases, both integer-only. Phase one computes the gross gain of a cycle from the three L1 reads:

// gain = Π_{sell legs} bid / Π_{buy legs} ask, mantissas at each leg's tick exponent
let scale = pow10(exp.unsigned_abs()).ok_or(SkipReason::Overflow)?;
if exp >= 0 { num = num.checked_mul(scale)?; } else { den = den.checked_mul(scale)?; }
let q = num / den;
let r = num - q * den;
let ppm = q * PPM + (r * PPM) / den;   // parts per million, 1_000_000 = break-even

Multiply the bids of the sell legs, multiply the asks of the buy legs, fix up the exponent difference with a power of ten, and divide once with the remainder trick so the intermediate never exceeds the denominator times a million. A cycle is a candidate when gross gain clears three legs of fees plus the emit threshold, with the additive fee bound chosen looser than the exact (1 − f)³ so phase one never discards something phase two would accept. Cost per cycle: three seqlock reads, up to six multiplications, one division.

Phase two runs only for candidates. Amounts are u128 in units of 10⁻⁸ of an asset, exact for Binance's eight-decimal precision. For each start asset present in the cycle it computes the capacity of each leg, the amount of input that would fully consume the displayed size, converts each back to the start asset by inverting the preceding legs, takes the minimum, caps it with the notional limit, and then walks that amount forward through the three legs with the exchange's lot rounding, minimum quantity and minimum notional applied. Leftover from the first leg is returned; dust stranded in intermediate assets is counted as lost. The best net result per cycle goes into a single-producer, single-consumer ring to the service side, and a full ring increments a drop counter instead of blocking the thread. That counter is zero in every run I have.

What it costs

Three runs stand for the rest. The first two are the keyless JSON feed from my desk, every pair, five connections; the third is SBE from an instance next to the exchange.

The machine is nowhere near busy. The cost of "every pair at once" is the hub symbols: one USDT/TRY update re-evaluates about 600 cycles and costs 55–63 µs at the median, and isolating those on their own thread is what keeps every other pair at a microsecond. Pinning the threads moved the thin-pair p99 from 10–27 µs to 7–16 µs and removed the 880 µs outliers, which were scheduler noise, not code. The 15-minute run also caught Binance closing four of the five connections within two seconds, a server-side event; all four were back within 1–2.3 s with the subscriptions restored and nothing dropped, which is the reconnect path earning its keep without me watching.

Measuring is part of the engine

Every hot thread keeps per-stage histograms, decode, decision, total and, when the host clock is synced, staleness against the exchange's eventTime, plus a few dozen counters written with plain relaxed stores by their single owner. A service thread snapshots them once a second for the terminal UI or a log line, and a JSON report is written on exit. None of this costs the hot path more than a couple of stores per event, and all of it was decided before the first line of the decoder, because the questions I needed answered were about the engine's view of the market, not about its throughput.

That turned out to matter more than any optimization. With chrony on the host, the staleness histogram said, from the first run in Tokyo, that a quote reached the engine 0.9–1.0 ms after the exchange stamped it, with decode and decision adding single-digit microseconds. The tails were theirs. I could stop tuning and start asking what the market does inside that millisecond, which is the subject of the next post.

The tests that keep it honest

The decoders have encoders, written only for tests and fixtures, so every layout claim in the table above is exercised by a round trip. The seqlock has the torn-read test. The evaluator has unit tests on hand-built cycles with known gains, lot sizes and minimum notionals, including the degenerate cases: a zero side, a capacity that rounds to nothing, an exponent sum that would overflow. The framer is tested over an in-memory stream with server frames of every length class, fragmented messages, and masked frames that must be rejected. And one integration test stands up a local mock WebSocket server, drives the whole pipeline through it with scripted frames, and checks the opportunities that come out the other end. 173 unit tests and the integration tests run without network in a few seconds, and they have caught every regression I made while the measurement work kept changing the engine underneath.

What I would do differently

Two things. I would make reconnects make-before-break from the start: open the new connection, subscribe, confirm, then close the old one. The close-then-reopen path loses one to two seconds of a connection's symbols, which is fine for measurement and not fine for anything that trades. And I would record the raw feed to disk from day one. The two-hour recorder I wrote later, in the same style, cost 1.2 % of a core and 9 MB of memory to capture 1.85 million frames, and it answered more questions in an evening than the live engine had in a week. A feed you can replay is worth more than a feed you can only watch.