KYC webhooks arrive out of order, so rank the decisions

An identity verification provider does not hand you a status. It hands you a stream of webhook events, and that stream arrives out of order, duplicated, and occasionally after one of your own compliance reviewers has already decided the case by hand. If your integration does user.kyc_status = payload['status'] on every callback, you will eventually flip a rejected applicant back to approved. You will not learn about it from your logs. You will learn about it from an audit.

I have been integrating KYC and KYB providers into payment and lending products for years, along with the real-time moderation tooling used by the humans on the other side of the queue. The bug always has the same shape, and the fix is always the same: stop trying to reconstruct a timeline you never had.

Why receipt order tells you nothing

Four independent things break ordering, and they compound:

• Provider clocks. Timestamps commonly land at second granularity, and a document check plus a liveness check finishing in the same second is not rare, it is the normal case.
• At-least-once delivery. Your endpoint returns a 500 once, or times out, and the same event is redelivered minutes later — after the events that logically follow it.
• Parallel pipelines on their side. Document extraction, face match, and watchlist screening are separate workers. They emit transitions concurrently, and nobody sequences them for you.
• Your own reviewers. A human acts in your database while the provider pipeline is still running.

So delivery order is meaningless, emission order is partly unknowable, and provider timestamps are not a total order. What you actually have is a set of decisions, each with a source and a meaning. A set does not need ordering. It needs precedence.

Store the events, decide separately

The first move is to stop letting the webhook handler own the status. It owns one thing: appending a fact.

create table kyc_event (
    id             bigserial   primary key,
    application_id uuid        not null references kyc_application(id),
    source         text        not null,   -- 'provider' | 'reviewer'
    external_id    text        not null,   -- provider event id, or internal action id
    decision       text        not null,   -- a value from the lattice below
    payload        jsonb       not null,
    received_at    timestamptz not null default now(),
    unique (source, external_id)
);

That unique constraint is the entire idempotency story. At-least-once delivery becomes exactly-once processing, enforced by the database rather than by a Redis key with a TTL that you will one day tune wrong.

def ingest(conn, application_id, source, external_id, decision, payload):
    with conn.cursor() as cur:
        cur.execute(
            '''
            insert into kyc_event
                (application_id, source, external_id, decision, payload)
            values (%s, %s, %s, %s, %s)
            on conflict (source, external_id) do nothing
            ''',
            (application_id, source, external_id, decision, Json(payload)),
        )
        if cur.rowcount == 0:
            return False  # duplicate delivery, already recorded

    reproject(conn, application_id)
    return True

Return 200 in both branches. A provider that receives a non-2xx will retry, and retrying a duplicate is pure noise that eventually buries the deliveries you actually care about.

One detail worth insisting on: reviewer actions go into the same table, with the reviewer action id as external_id. An event log that covers only the provider is half an audit trail, and it is the half that never explains the interesting cases.

Precedence, not chronology

Now define what outranks what. This is not a state machine over time — it is a lattice over outcomes.

RANK = {
    'started':            0,
    'pending_documents': 10,
    'in_review':         20,
    'approved':          30,
    'rejected':          40,
    'blocked':           50,   # watchlist / AML hit
}

Read it as: an application's visible status is the highest-ranked decision anyone has ever made about it. approved sits below rejected because a late-arriving approval from a parallel pipeline must never erase a rejection. blocked sits above everything because a screening hit is not negotiable by a document check that finished afterwards.

The projection is then a pure function of the event set:

def project(decisions):
    return max(decisions, key=lambda d: RANK[d], default='started')

And the write is guarded by rank rather than by time:

def reproject(conn, application_id):
    with conn.cursor() as cur:
        cur.execute(
            'select decision from kyc_event where application_id = %s',
            (application_id,),
        )
        status = project([row[0] for row in cur.fetchall()])
        cur.execute(
            '''
            update kyc_application
               set status = %s, status_rank = %s, updated_at = now()
             where id = %s
               and status_rank < %s
            ''',
            (status, RANK[status], application_id, RANK[status]),
        )

The status_rank < guard is what makes concurrency boring. Two webhook workers can process two events for the same application at the same time, in either order, more than once, and the row converges to the same value. No advisory locks, no serializable transactions, no ordering queue keyed by application id. Monotonicity does the work.

A human cannot un-reject; a human opens a new case

The obvious objection: applicants do get rejected for a bad document scan and then send a good one. Under a monotonic lattice, nobody can walk the status back down — which is exactly the property compliance wants, and exactly the property product does not.

Both are satisfied by moving the reversal up a level. A reviewer does not edit the decision; they open a new application row for the same user, with an incremented attempt number. The previous case stays terminal forever, with its evidence attached. The user-facing status is the status of the latest case.

def reopen(conn, user_id, previous_application_id, actor_id, reason):
    with conn.cursor() as cur:
        cur.execute(
            '''
            insert into kyc_application (user_id, attempt, status, status_rank)
            select user_id, attempt + 1, 'started', 0
              from kyc_application
             where id = %s
            returning id
            ''',
            (previous_application_id,),
        )
        new_id = cur.fetchone()[0]

    ingest(conn, new_id, 'reviewer', f'reopen:{actor_id}:{previous_application_id}',
           'started', {'reason': reason})
    return new_id

Nothing is ever overwritten. When someone asks in six months why a particular user is allowed to move money, the answer is a select, not an archaeology project.

What this buys you later

Because status is a projection rather than an accumulated side effect, three otherwise painful operations become routine. You can change the lattice — say, you introduce a manual_hold decision — and backfill by re-running reproject over the affected applications. You can replay a provider's event history after an outage without worrying about what it does to users who have since been decided. And you can diff your projection against the provider's own view of the case in a nightly job, which is how you discover that a webhook was silently dropped weeks ago.

That last one matters more than it sounds. Missing events are invisible in a last-write-wins design: the row simply holds a stale value that looks plausible. With an event table, a reconciliation query has something to compare against.

What I would do differently

My first version of this had a status column and a guard on the provider's updated_at. It survived exactly until two events shared the same second and the wrong one won. I patched it with a rule that terminal states cannot be overwritten. Then I patched that with a special case for screening hits. At that point I had a precedence lattice implemented as scattered if statements inside a webhook handler, which is the worst of both designs: the semantics of a lattice with none of the auditability.

So: write the rank table first, even if you only have three states. Put reviewer actions in the same event stream from day one. And never guard a state transition on a clock you do not control.

The shape generalises well beyond KYC. Any time you consume at-least-once events from a source that runs work in parallel — payment processor callbacks, chain reorg notifications, carrier tracking updates — the same three moves apply: append every decision with a natural idempotency key, define what outranks what, and make the visible state a projection you can recompute at any time.

Originally published on polycratia.com — where I write about payment systems, crypto rails and marketplace backends.