Financial services

An Exchange That Survived Its First Listing Day

A digital asset venue needed its first live listing to behave as a controlled operational event rather than a traffic experiment. The work concentrated on deterministic order state, custody boundaries, reconciliation and observable failure modes around the matching engine.

Engagement
Exchange infrastructure
Duration
Around two quarters
Team
Backend and platform engineers with a delivery lead
A wireframe drawing of a steep surge in a stepped column profile striking a reinforced frame held by load-bearing struts.

This account is anonymised. The client is described by sector rather than by name, and outcomes are described in words rather than figures — no percentage or saving is quoted because none can be independently evidenced here.

Before the work

The matching engine was only one boundary

The venue had a functional matching component, but production readiness depended on everything around it. API requests, account balances, deposits, withdrawals, market-data publication and operational controls each maintained state with different timing and failure behaviour. A successful match was not useful if a retry could reserve funds twice, if a websocket client saw trades before the account ledger did or if operations could not explain why a withdrawal remained pending. The first listing would also compress user activity and administrative change into the same period, exposing assumptions that had held only under orderly test traffic.

The engagement avoided claims that distributed components could be made simultaneously atomic. Instead it identified invariants and authoritative records, then designed explicit transitions between them. The matching engine owned order priority and execution. An immutable double-entry ledger owned asset balances. Custody adapters reported observed chain or provider state without becoming the accounting record. Durable events connected these domains through idempotent consumers, sequence checks and reconciliation. Listing configuration, market opening and emergency controls were treated as governed operations with reviewable preconditions, not database edits performed while the venue was live.

Constraints

What shaped the build

Irreversible external settlement

Blockchain and custody actions could not be rolled back like database transactions, so intent, submission and confirmation required distinct states.

Bursting market activity

Order entry and market-data paths needed bounded backpressure and degradation behaviour during concentrated demand.

Asset-specific behaviour

Precision, confirmation, fee and maintenance rules varied by asset and had to be configuration with controlled change.

Operational approval

Market opening, withdrawals and protective controls required separation of duties and an immutable decision trail.

Approach

How the engagement ran

  1. 01

    Specify invariants

    We defined order, reservation, execution, ledger and settlement states, including who owned each transition and how duplicates were recognised.

  2. 02

    Harden service boundaries

    Durable events, idempotency keys, transactional outboxes and sequence-aware consumers replaced timing assumptions between components.

  3. 03

    Build operational controls

    Listing configuration, market status, withdrawal policy and incident actions received approvals, validation and audit history.

  4. 04

    Exercise failure

    Load and fault scenarios covered delayed consumers, duplicated messages, stale market data, custody interruption and partial service recovery.

  5. 05

    Rehearse the listing

    The opening sequence, readiness evidence, communications and stop conditions were run against production-like configuration before authority transferred.

01

Keep order state deterministic

Order submission began with authentication, market status and syntactic validation, followed by a funds reservation against the account ledger. Client-supplied idempotency keys allowed a timed-out request to recover its original result instead of creating another order. Accepted commands entered a sequenced stream consumed by the matching component, whose deterministic state transitions produced executions, cancellations and rejections. The API did not infer final order state from whether an HTTP response had returned. Clients could query the durable order record, and internal services consumed ordered events with gap detection. A missing sequence stopped that partition from inventing continuity and raised an operational exception.

Balance reservation and execution posting were connected without pretending to share a transaction. The ledger recorded reserved and available positions through balanced entries, while a transactional outbox published committed changes. Execution consumers posted transfers using the execution identifier as an idempotency key, then released any residual reservation when an order closed. Reconciliation compared engine orders and executions with reservations and ledger entries, exposing orphaned or delayed state by source identifier. Corrections used compensating ledger entries and linked operational decisions; balances were never repaired by updating a current total. This provided a history capable of explaining both ordinary trading and recovery after an interrupted consumer.

02

Separate custody observation from accounting truth

Deposits moved through observed, attributed, pending and available states. Chain listeners and custody providers could resend notifications or revise confirmation status, so adapters normalised external events and retained their source identity. Crediting occurred through an idempotent ledger event after policy conditions were met, not whenever a webhook arrived. Reorganisations, unsupported transfers and ambiguous destination data entered explicit exception paths. The displayed balance came from the ledger, with external wallet state reconciled against it as a separate control. This prevented provider availability or eventual chain indexing from determining whether internal accounting history existed.

Withdrawals began as controlled intents. Validation, account reservation, policy checks, approval where required, signing submission and external confirmation were separate transitions with immutable timestamps and actors. A timeout after submission did not authorise automatic resubmission until the provider or chain had been queried using the original reference. Fee treatment and returned transactions generated ledger entries rather than editing the withdrawal amount in place. Custody credentials stayed behind a narrowly scoped adapter and were not available to general exchange services. Operational tooling exposed stuck states and safe next actions without revealing signing material or offering a generic force-complete button.

03

Publish market data with sequence and backpressure

Market data was derived from the authoritative engine event stream. Snapshots carried a sequence position, and incremental updates allowed clients to detect a gap and resynchronise rather than continue displaying an invalid book. The websocket tier bounded per-connection buffers and disconnected clients that could not consume safely; it did not allow slow subscribers to exhaust shared memory. Public trade, ticker and book views could be degraded independently from authenticated order entry. Internal monitoring compared publication lag with engine position, making stale but apparently healthy connections visible before users reported contradictory screens.

Load testing modelled behaviour rather than a single request rate. Scenarios included concentrated order placement, cancellation waves, reconnecting market-data clients, repeated idempotent requests and consumers recovering a backlog. Fault injection delayed event handling, interrupted the custody adapter and restarted stateful components from snapshots plus logs. Tests asserted invariants after recovery: no unexplained ledger imbalance, no execution applied twice, no order progression across a sequence gap and no withdrawal resubmitted because of an ambiguous response. Capacity decisions followed the measured saturation point of each bounded queue, with explicit rejection or backpressure before the component became unavailable in an uncontrolled way.

04

Treat a listing as a governed change

A market definition contained base and quote assets, precision, minimum increments, fee rules, status and external identifiers. Configuration was validated as a coherent version, reviewed and promoted through environments rather than patched directly in production. Readiness checks confirmed custody support, ledger accounts, deposit and withdrawal policy, market-data channels, monitoring and operational ownership. The market progressed through disabled, cancel-only and trading states under authorised commands. Opening was therefore reversible at the venue-control level even though already accepted executions remained durable facts. Protective action stopped new risk without rewriting completed activity.

The listing rehearsal exercised the same runbook and permissions intended for live operation. Dashboards were organised around invariants and queues: event lag, sequence gaps, unmatched reconciliation items, wallet exceptions and rejected commands. Alerts linked to a specific operational state and documented response rather than reporting undifferentiated infrastructure noise. During opening, decision records captured configuration version, readiness acceptance and any control change. Afterwards, reconciliation ran across orders, executions, fees, ledger balances and custody observations before routine operation resumed. The listing held because the system had defined how to refuse, pause, replay and explain work, not because every dependency was assumed to remain fast and available.

Outcomes

What changed

  • Order retries resolved to durable state instead of risking duplicate reservations.
  • Every execution could be followed into balanced ledger entries and account movements.
  • Custody interruptions produced explicit pending states rather than inferred completion.
  • Market-data clients could detect gaps and recover from a known sequence position.
  • Listing controls became reviewed configuration and authorised commands instead of live database changes.

Stack

  • Node.js
  • TypeScript
  • PostgreSQL
  • Redis
  • Kafka
  • Kubernetes
  • OpenTelemetry
FAQ

Common questions

Is the matching engine the main part of building an exchange?

No. Ledger integrity, custody integration, order gateways, market data, reconciliation and operational controls determine whether matching can be run safely as a service.

How do exchange APIs prevent duplicate orders?

A client idempotency key is bound to the authenticated request and its durable result. Retries return that result, while downstream consumers also deduplicate against stable event identifiers.

How are exchange balances reconciled with custody?

The internal double-entry ledger remains the accounting record and custody state is an external control population. Deposits, withdrawals, fees and exceptions are bridged by stable source references.

What happens when market data falls behind?

Sequence-aware clients detect gaps and obtain a fresh snapshot before applying more increments. Bounded buffers and deliberate disconnection prevent slow consumers from destabilising the publication tier.

Describe your version of this.

The constraints are never identical, and that is usually where the interesting engineering is. Tell us what yours are.