DEV Community: Luna AI

Designing an AI Evidence Gateway: Durable WAL, Portable MMR Proofs, and Bounded Formal Checks

Luna AI — Wed, 26 Aug 2026 15:05:00 +0000

AI gateways are often discussed as routing layers: authenticate a caller, apply policy, forward a request, and record what happened. For higher-assurance deployments, the harder engineering question is what evidence remains after a request completes—and exactly what that evidence proves.

Aegis Latent Core is an AI Governance and Evidence Gateway for governed LLM traffic. Its current source implements provider-aware request controls, bounded streaming redaction, durable evidence records, portable Merkle Mountain Range (MMR) inclusion proofs, Python and TypeScript integrations, and narrowly scoped formal checks.

This article walks through those mechanisms and, just as importantly, their boundaries.

Version note: GitHub currently exposes a Release labeled v4.0.1, while the source anchors, all ten named Release assets, and the published SDKs identify version 4.0.0. The v4.0.1 ref points directly to a commit, making it a lightweight tag. Treat v4.0.1 as the GitHub Release label—not as evidence that the artifacts were rebuilt as 4.0.1, that the tag is signed, or that the tag-triggered publication workflows produced the public objects.

The request and evidence path

At a high level, Aegis sits between an application and a configured model provider. The non-streaming and streaming paths share admission controls, but they cross the evidence boundary differently.

Client application
    |
    | request
    v
Aegis gateway
    |-- authenticate caller and assign request identity
    |-- enforce body bounds and canonicalize input
    |-- apply WAF, egress, session, and rate-limit controls
    |
    | admitted request
    v
Configured upstream model provider
    |
    | response or terminal error
    v
Aegis gateway
    |
    |-- non-streaming ---------------------------------------|
    |   canonicalize outcome                                |
    |   hash and commit evidence to authoritative JSONL WAL |
    |   flush + fsync                                       |
    |   return governed response                            |
    |                                                       |
    |-- SSE streaming --------------------------------------|
        incrementally sanitize canonical events             |
        emit through a bounded, byte-accounted queue        |
        hash the exact bytes emitted                         |
        commit one terminal summary                          |
        emit the protocol terminal marker only after commit |
                                                            v
                                                proof lookup / audit views

For non-streaming traffic, the response crosses the durable evidence gate before it is returned. For admitted Server-Sent Events (SSE), sanitized non-terminal events can be emitted incrementally. Initial evidence and proof status is therefore pending-terminal; the success terminal marker is withheld until the terminal summary commits.

The replay authority is an append-only JSONL WAL at the configured storage path. It stores chain linkage, request and response hashes, portable proof metadata, signature metadata, and request identity. Calling fsync means the process asked the operating system to synchronize the descriptor; it does not by itself prove power-loss behavior, replicated-volume durability, immutable retention, or external custody. Those remain deployment questions.

Where Rust fits—and where it does not

The repository includes an optional native RustWal. When the extension is available, it receives an auxiliary copy of terminal streaming frames in a bounded memory-mapped, CRC32-framed segment at <wal_path>.stream.rwal.

That distinction matters: the Rust segment is not the replay authority. The authoritative record is the fsynced JSONL WAL. Describing the architecture as a purely Rust-backed ledger, a zero-copy path, or a latency-free evidence layer would overstate the implementation.

Bounded streaming redaction

Streaming privacy controls have a chunk-boundary problem. An identifier can begin in one event and finish in the next. Aegis addresses supported identifier forms with a finite character holdback rather than buffering an entire response.

The stream path also applies per-admitted-stream bounds to queue bytes, queue items, event size, cumulative output, preview retention, de-identification window, and duration. SHA-256 covers the exact emitted bytes, and the implementation commits one terminal outcome. On byte, event, duration, cancellation, incomplete-upstream, or upstream-error paths, it closes the upstream iterator where applicable and omits the success terminal marker.

These are useful, testable controls—not universal de-identification. Aggregate retained memory still scales with admitted concurrency, so deployment-level admission and concurrency controls remain necessary.

Portable MMR inclusion proofs

Aegis uses the aegis-mmr-inclusion-v1 format so a client can verify that a disclosed leaf digest is included relative to a supplied root without possessing the gateway's in-memory MMR state.

The construction is deliberately specific:

A raw leaf becomes the lowercase hexadecimal SHA-256 digest of its bytes.
An internal node is SHA-256 over the ASCII concatenation of the left and right lowercase hexadecimal digests.
Current peaks are ordered by descending mountain height.
The displayed root is SHA-256 over the ASCII concatenation of those ordered peak digests.
Verification checks the leaf index and count, mountain boundaries, path length, sibling direction, complete peak set, containing peak, and final root.

For a durable non-streaming response, proof data can be returned in X-Aegis-MMR-* headers. An SSE response cannot update its initial headers after streaming begins, so its proof must be retrieved from the authenticated proof link after terminal commit.

The trust anchor is crucial. A verifier must pin or otherwise obtain the root through an independent trust policy. Copying a root from the same untrusted response does not make it independent. A valid proof establishes inclusion relative to that root; it does not establish that the source event is true, that timestamps are externally trusted, that storage is immutable, or that the evidence is legally admissible.

Python quickstart

The published Python distribution is aegis-latent-sdk==4.0.0. The install name uses hyphens; the import namespace is aegis_sdk. Python 3.11 or newer is required.

python -m pip install 'aegis-latent-sdk[openai]==4.0.0'

import os
from aegis_sdk.openai import OpenAI

client = OpenAI(
    aegis_api_key=os.environ["AEGIS_API_KEY"],
    gateway_url=os.environ["AEGIS_GATEWAY_URL"],
    tenant_id=os.environ["AEGIS_TENANT_ID"],
)

response = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[{"role": "user", "content": "hello"}],
)

The package supplies tested OpenAI and Anthropic subclasses within declared dependency ranges. Subclassing does not make every vendor route available through the gateway. Native Anthropic Messages ingress requires the gateway to be configured with AEGIS_PROVIDER=anthropic.

For automatic non-streaming proof verification, callers can set verify_proof=True and provide a pinned trusted_mmr_root. Streaming proofs are retrieved after terminal commit instead of being completed in the initial headers.

TypeScript quickstart

The npm package is aegis-latent-sdk@4.0.0. The current package declares Node.js 18 or newer and provider SDKs as peer dependencies.

npm install aegis-latent-sdk@4.0.0 openai@^6.49.0

import OpenAI from "openai";
import { openAIGatewayOptions } from "aegis-latent-sdk";

const client = new OpenAI(openAIGatewayOptions({
  aegisApiKey: process.env.AEGIS_API_KEY!,
  gatewayUrl: process.env.AEGIS_GATEWAY_URL!,
  tenantId: process.env.AEGIS_TENANT_ID!,
}));

Proof verification is available as a separate operation:

import {
  parseInclusionProof,
  verifyInclusionHash,
} from "aegis-latent-sdk/proof";

const proof = parseInclusionProof(untrustedJson);
const valid = await verifyInclusionHash(
  leafHashHeader,
  proof,
  pinnedRoot,
);

The verifier uses Web Crypto-compatible primitives. The package also offers provider-native integration surfaces, but compatibility remains bounded to the versions, routes, and behaviors covered by its tests.

What the formal gate checks

The repository's formal gate combines two SMT-LIB checks, one Lean 4 theorem, and three finite TLA+/TLC models. Its declared checks include token-bucket arithmetic, per-stream retained-memory arithmetic, a durable-before-emission phase theorem, append-only ledger prefixes, commit-before-emission in a finite request model, and session-to-ledger binding.

This is stronger and more precise than saying “formally verified.” The artifacts verify their stated formulas and bounded abstractions. They do not provide a machine-checked refinement proof connecting every Python or Rust transition—or the operating system and target filesystem—to those models. In particular, the lifecycle abstraction does not model every incremental SSE event as withheld; it models committed outcomes and the terminal transition.

From a source checkout, the formal entry point is:

bash scripts/verify_formal_artifacts.sh

Treat a successful run as evidence about those named artifacts and bounds, not as certification of the whole product.

Run the audit dashboard locally

The repository includes a read-only dashboard for health, retained ledger views, proof lookup, current metrics, and bounded forensic export. The dashboard uses the sibling TypeScript SDK through a local file dependency, so build that source package first even though an SDK is now published on npm.

git clone https://github.com/JuanLunaIA/aegis-latent-core.git
cd aegis-latent-core
git checkout 6469904380218584ae0b5221334bc9a46500f5ba

cd sdk/typescript
npm ci
npm run build

cd ../../dashboard
npm ci
export AEGIS_PRIMARY_BASE_URL='https://aegis.internal'
export AEGIS_DASHBOARD_API_KEY='read-only-audit-token'
npm run build
npm start

For interactive development, run npm run dev instead of the final build/start pair. Use a dedicated least-privilege audit token and place the dashboard behind an authenticated reverse proxy. Although the UI does not mutate ledger records, the Forensics page can request a bounded evidence ZIP and requires audit:export; treat that as a sensitive export operation.

The dashboard does not fabricate missing data. An unavailable endpoint remains unavailable rather than becoming a zero or a demo value. Its MMR page can run the shared verifier in the browser, but a root returned by the same gateway is still not an independent trust anchor.

Evaluate the evidence, not the adjectives

A useful review starts with narrow questions:

Which WAL is authoritative, and what happens on append or fsync failure?
Which SSE bytes are hashed, and when does proof status leave pending-terminal?
Where does the verifier obtain an independently trusted MMR root?
Which provider routes and dependency ranges are covered by integration tests?
Which formal property, state-space bound, and implementation gap apply to each claim?
Which deployment controls provide identity, durable storage, key custody, retention, ingress, and recovery?

Aegis implements technical controls and produces structured cryptographic evidence under declared conditions. It is not, by repository evidence alone, a compliance certification, a legal-admissibility ruling, a production SLO, a WORM guarantee, or proof of external immutability.

Source and packages

If you review the project, feedback on the authoritative/auxiliary WAL boundary, portable-proof trust anchoring, and streaming terminal semantics would be especially useful.

Bridging Python and Rust: Mitigating GIL Contention in a High-Throughput LLM Gateway

Luna AI — Mon, 22 Jun 2026 02:18:52 +0000

Bridging Python and Rust: Mitigating GIL Contention in a High-Throughput LLM Gateway

When building Aegis, an open-source OpenAI-compatible governance proxy, we made a core architectural decision: use Python (FastAPI/ASGI) for rapid development and API adaptability, but offload high-performance cryptography, Write-Ahead Logging (WAL), and Merkle Mountain Range (MMR) operations to a compiled Rust extension (aegis_rust_v2) via PyO3 and Maturin.

However, mixing Python’s asynchronous event loop with Rust's multi-threaded Tokio runtime led us directly to a classic systems engineering wall: GIL (Global Interpreter Lock) contention.

Here is a deep dive into the architecture, the performance tradeoffs, and how we engineered a two-path model to keep hot-path latency under 2.5 microseconds.

The Two-Path Execution Model

In LLM governance, every microsecond of added proxy latency is a penalty for the client application. To achieve zero client-visible audit wait, Aegis splits the request path:

        ┌──────────────────────── HOT PATH (Awaited) ───────────────────────┐
client →│ smuggling guard → auth → WAF → rate-limit → adapter → forwarder →  │→ upstream
        └───────────────────────────────────┬───────────────────────────────┘
                                             │ _spawn_background() (~2.4 µs)
                                             ▼
        ┌──────────────────── BACKGROUND PATH (asyncio.create_task) ─────────┐
        │ ResponseAnalyzer → CryptographicAuditLedger → MMR → Write-Ahead Log│
        └────────────────────────────────────────────────────────────────────┘

The ASGI server returns the upstream JSON response to the client before the auditing, Shannon token entropy analysis, and cryptographic hashing take place.

The only work done on the hot path is scheduling the task. In our benchmark environment (Intel Xeon @ 2.80 GHz, 4 cores), this scheduling block (asyncio.create_task + background set tracking + Prometheus gauge updates) costs only 2.43 µs p50 and 6.78 µs p99.

Accelerating the Audit Path with a Rust MMR

Once the background task is spawned, it hands over data to the CryptographicAuditLedger. This is where Rust shines.

Each committed transaction appends a leaf to a growing Merkle Mountain Range (MMR)—an append-only logarithmic accumulator that provides inclusion and consistency proofs without needing the massive rebalancing overhead of a classic balanced binary Merkle tree.

In Python, the leaf hashing looks like this:

# Pure Python fallback
def add_leaf(self, leaf_hash: bytes) -> bytes:
    self.leaves.append(leaf_hash)
    # Merging peaks involves allocating many small bytes objects
    # causing measurable GC pressure at scale...

By binding Rust via PyO3, we run the inner-loop tree accumulation natively without allocations per node:

// aegis_rust_v2/src/mmr.rs
#[pyclass]
pub struct MmrAccumulator {
    peaks: Vec<Option<[u8; 32]>>,
    count: usize,
}

#[pymethods]
impl MmrAccumulator {
    pub fn add_leaf(&mut self, leaf: &[u8]) -> PyResult<String> {
        // Direct, zero-allocation peak merging using native SHA-256
    }
}

This Rust acceleration layer delivers a stable 3.01x to 3.34x speedup over the pure Python baseline:

N (leaves)	Python (leaves/s)	Rust (leaves/s)	Speedup
100	332,460	958,510	2.88×
1,000	292,050	814,000	2.79×
10,000	250,650	760,260	3.03×
100,000	212,180	709,240	3.34×

Hitting the GIL Contention Wall

Despite the speedups, we noticed an anomaly during concurrent loopback performance sweeps (GET /health hitting the entire ASGI, WAF, rate-limiting, and live ledger check stack):

Concurrency 1: 650 RPS | 1.49 ms p50 | 35.7% CPU
Concurrency 4: 902 RPS | 4.05 ms p50 | 43.1% CPU
Concurrency 32: 339 RPS | 65.2 ms p50 | 18.7% CPU
Concurrency 128: 246 RPS | 297.6 ms p50 | 13.8% CPU

Notice how past $c=4$, throughput drops and latency climbs exponentially, yet the CPU utilization decreases.

This is event-loop head-of-line blocking caused by GIL contention [INFERENCE]. Every time the Python ASGI loop yields to coordinate an event or a lock, if the Rust threads (running the background Tokio pool or PyO3 cryptographic calls) hold the GIL, the Python loop stalls. Even though Rust is extremely fast, the cost of acquiring and releasing the GIL via PyO3's FFI interface scales with concurrency.

The Architectural Implication

This benchmark gave us an empirical design answer: scale out, not up per worker.

Instead of piling client concurrency onto a single Python process and relying on massive thread-pools, the optimal deployment strategy for Aegis is:

Run one Uvicorn worker process per physical core.
Restrict container CPU limits to match the worker count exactly (avoiding CFS throttling).
Front with a load balancer (e.g., NGINX, HAProxy, AWS ALB) using tenant-affinity hashing.

By pinning workers and keeping concurrency low per process, we keep the ASGI event loop completely clear of FFI contention while maintaining full audit durability.

Conclusion and Open Source

Aegis is fully open-source under the AGPLv3 license. If you are building generative AI integrations in highly regulated sectors (or just want to play with PyO3, Maturin, and cryptography), check out our code:

👉 GitHub: https://github.com/juanlunaia/aegis-latent-core

👉 Visualizer Dashboard: https://github.com/juanlunaia/aegis-latent-core/tree/main/tools/visualizer

I’m a 22-year-old student from Argentina, and I’m actively seeking feedback on this FFI architecture. If you've solved similar ASGI/PyO3 threading bottlenecks, I would love to hear how you did it!