DEV Community: Jonathan Murray

I Counted the Attack Vectors in Our AI Stack and Now I Can't Sleep

Jonathan Murray — Wed, 26 Aug 2026 17:29:30 +0000

so a while back i did something i do not recommend before bed.

i counted the vendors in a normal production AI stack. not a weird stack. not a bad team's stack. the stack everyone builds, the one in every architecture diagram on this site.

model gateway. vector database. memory service. rag framework. embeddings provider. orchestration. observability. two or three model providers.

that's eight. for one app.

and here's the first thing that got me: nobody approved that stack. not one person at any company, ever, sat down and said "yes, eight vendors, sounds right." it got approved one sprint at a time. a ticket here, a free tier there, a "we'll swap it later" that became load-bearing.

your architecture wasn't designed. it accreted. like a reef. or plaque.

the fun part: what each one actually is

security people have a phrase, "attack surface," and it sounds abstract until you translate each vendor into what it really is:

a model gateway is a standing API key in your CI. a vector database is a second copy of your data. a memory vendor is your prompts in someone else's logs. a rag framework is somebody's SDK running inside your process. observability is prompt logs leaving your network as the product working correctly.

read that last one again. the exfiltration is the feature. you pay for it monthly.

quick quiz you will fail

open two lists. one: every AI line item in your billing. two: every AI-related package in your lockfiles.

do they match?

they never match. i've asked a lot of teams. the answer is always no, and everyone laughs, and it's the nervous kind of laugh, because the gap between those two lists is software that touches your customer data and appears in exactly zero of your security reviews.

the gap is the attack surface. congratulations, you just did a security audit with grep.

"embeddings are just numbers though"

this is my favorite myth in all of AI, because it's so comforting and so wrong.

the theory: we embedded the text, so now it's a vector, vectors are anonymous math, we're fine.

the practice: embedding inversion is published research. morris et al. (2023, "text embeddings reveal (almost) as much as text") reconstructed the exact input text from its embedding 92% of the time for short inputs. not "similar vibes." the actual text.

so your vector database is not a bag of anonymous numbers. it's a second, queryable database of your source data, with its own API key, sitting in someone else's cloud.

now the aha that hurts: go ask whoever maintains your data inventory whether the vector store is on it.

it's not on it. it's never on it. you have a database of customer data that your own compliance team doesn't know exists, and you built it on purpose, with a jira ticket.

deletion theater

follow one user message through the stack. it hits the gateway (logged). gets embedded (copy two, the vector db). the memory service persists it (copy three). observability captures the full trace (copy four).

one message. four systems. four retention policies you've never read.

now legal walks over: "customer invoked their right to erasure, delete their data."

you cannot do that. what you can do is file four vendor tickets and hope. under gdpr this is the difference between "we called the delete endpoint and logged the result" and "we asked four companies nicely." one of those survives an audit. guess which one you have.

i call this deletion theater. everyone has a retention policy. almost nobody has a delete button that actually reaches all four copies.

deletion you can't prove is deletion you don't have. put it on a poster.

meanwhile, your dependencies

your rag framework is not one dependency. it's a tree. hundreds of packages, written by strangers, pinned loosely, executing in the same process that holds your database credentials.

this isn't hypothetical doom. langchain shipped a remote code execution bug in its math chain (CVE-2023-29374). pytorch's nightly build got hit by dependency confusion in 2022. owasp made supply chain a whole category in its LLM top 10 because this keeps happening.

the framework's security posture is your security posture. you married it. you just didn't have a ceremony.

and the stack you didn't even choose

everything above is the stack you deployed on purpose. there's another one.

it's your employees pasting contracts, code, and customer records into whatever consumer chatbot they like, on their personal accounts, right now, while you read this.

your company probably "banned" this. here's the aha: the ban is working as a compliance document and failing as a control. people don't stop using tools that make them faster. they stop telling you.

the only fix i've seen work is replacement, not prohibition. one sanctioned surface, the models people actually want, behind sso, logged. you can't firewall your way out of shadow AI. you can only out-compete it.

so what do you actually do

the counterintuitive answer, and the whole point: the fix is not more security. it's less stack.

every security tool you bolt onto an eight-vendor sprawl is a ninth vendor. the arithmetic only moves in one direction: fewer systems holding data, fewer standing keys, fewer egress paths. concretely, in priority order:

fewer copies. every system holding prompts or embeddings is a copy. count them, then make the number smaller.
one egress door. all model traffic through one brokered path where policy lives. no SDK in app code dialing providers directly.
one control plane, your keys. one place to rotate, one audit trail that answers "which model saw which data." byok, so consolidating the plane never means handing over the keys.
deletion as an api call. if you can't prove deletion programmatically across every copy, see poster above. you can build all of this yourself with enough glue code and discipline. some teams genuinely do. it's a lot of glue.

the horse, as promised

this counting exercise is roughly why backboard exists. routing across 17,000+ models, memory (first on the LoCoMo and LongMemEval benchmarks, receipts on github), embeddings, rag, and stateful threads behind one api and one key. for enterprises it ships as a signed container inside your own cloud, behind the IAM and SOC and audit trail you already trust. egress to outside models is opt-in per workload, never standing. deletion is an endpoint, not a ticket.

the objection i get, and it's a fair one: "congrats, you built a single point of compromise." three answers. byok means the platform never holds your keys, so consolidating the control plane isn't consolidating the credentials. in the enterprise deployment the one door is your door, inside your perimeter. and memory is exportable and routing is model-agnostic, so the exit is always open, which is the only honest answer to lock-in.

also, you'll notice zero certification badges in this post. on purpose. we publish nothing we haven't earned. ask for the architecture, not the badge.

the whole post in four lines

eight vendors is eight keys, eight egress paths, eight retention policies, and eight reviews your security team owes.

your embeddings are your data. your vector db is a database. put it on the map.

deletion you can't prove is deletion you don't have.

and the fix is not more security. it's less stack.

if you want to tear this apart, the code is at github.com/Backboard-io and my dms are open. i counted. your turn.

Every Layer of Your AI Stack Is an Attack Vector. Count Them.

Jonathan Murray — Wed, 26 Aug 2026 17:17:03 +0000

From the team at Backboard.io. We build AI infrastructure, so we have a position here. We state it at the end, clearly labeled. Everything before that is just counting.

TL;DR

A typical production AI application runs on 6 to 9 separate vendors: a model gateway, a vector database, a memory service, a RAG framework, an embedding provider, an orchestration layer, an observability tool, and 2 to 3 model providers. Each vendor adds at least five things to your attack surface: a standing API key, an egress path out of your network, an SDK executing inside your runtime, a log store that fills up with prompts, and a subprocessor on your data processing agreement. Most teams never approved this stack as a whole. They approved it one sprint at a time. This post is about how to count what you have actually deployed, and what reducing it looks like.

What is an AI attack surface?

An AI attack surface is the sum of every point where an attacker could reach the data, credentials, or compute involved in your AI workloads. For a modern LLM application it has three parts: the model layer (the providers your prompts travel to), the data layer (every system that stores prompts, embeddings, memories, or retrieved documents), and the integration layer (every SDK, framework, and glue service with credentials to the other two).

The mistake most teams make is measuring only the first part. The model provider gets a security review. The seven services wrapped around it usually do not.

The stack you actually deployed

Here is the stack a typical team assembles for one production agent, and what each layer costs you in security terms.

Layer	What it does	What it adds to your attack surface
Model gateway	Routes requests to providers	A standing key in CI, often with org-wide scope
Vector database	Stores embeddings for retrieval	A second queryable copy of your source data
Memory service	Persists user and agent state	Your prompts in someone else's logs, under their retention policy
RAG framework	Chunking, retrieval, prompt assembly	A large SDK and its dependency tree executing in your runtime
Embedding provider	Turns text into vectors	Separate billing, separate breach notification clock
Orchestration	Chains tools and agents	Glue code holding credentials for everything else
Observability	Traces and evaluates LLM calls	Prompt and completion logs leaving your boundary by design
Model providers (2 to 3)	Inference	Standing egress to each, each with its own retention terms

Now multiply. Eight vendors means roughly eight API keys with their own rotation stories, eight egress destinations your firewall must allow, eight vendor security reviews (or eight reviews skipped), eight subprocessors added to your DPA, eight breach notification clocks that can start ticking independently, and eight dependency trees whose CVE feeds someone on your team should be watching.

None of these vendors is careless. That is not the point. The point is arithmetic: every additional system that holds a copy of your data or a credential to your systems is surface, no matter how well run it is.

Why vector databases are the layer nobody reviews

Vector databases deserve their own section because they carry the most misunderstood risk in the stack.

The common assumption is that embeddings are safe because they are "just numbers." The vector is treated as a one-way hash of the text. It is not. Embedding inversion is a published, reproducible attack class: Morris et al. (2023, "Text Embeddings Reveal (Almost) As Much As Text") demonstrated iterative reconstruction that exactly recovers 92% of short text inputs from their embeddings, and Song and Raghunathan (2020) showed embeddings leak both content and authorship. If your source text was sensitive, treat the vectors as sensitive. Full stop.

That reframes what a vector database is: a second, queryable copy of your source data, sitting in a different trust boundary, usually with its own API key, and frequently excluded from the data inventory your compliance team maintains.

Three questions to ask about yours today:

Access: does the vector DB key in your app config have read access to every namespace, or is it scoped per workload?
Inventory: does your data map list the vector store as a location where customer data lives? (If a regulator asks, "just numbers" is not an answer.)
Deletion: when a customer invokes their right to erasure, does your pipeline delete the embeddings, or only the source rows? That third question is where most stacks fail, which brings us to the multiplication problem.

The deletion problem: one prompt, four retention policies

Follow one user message through a stitched stack. It lands in the gateway's request log. It is embedded, so a derivative lands in the vector database. The memory service persists a version of it. The observability tool captures the full trace, prompt and completion included.

That is one piece of customer data in four systems, under four retention policies, behind four vendor APIs with four different deletion semantics. When legal asks you to delete a specific user's data, you cannot execute that. You can file four tickets and ask nicely. Under GDPR Article 17 and similar regimes, "we asked our subprocessors" is a much weaker position than "we called the delete endpoint and logged the result."

Deletion you cannot prove is deletion you do not have.

The supply chain you inherited

The integration layer has its own history. LangChain shipped a remote code execution vulnerability in its math chain (CVE-2023-29374). The PyTorch nightly build was compromised through dependency confusion in December 2022. The OWASP Top 10 for LLM Applications lists supply chain vulnerabilities as a category precisely because the AI ecosystem moves fast and pins loosely.

Every framework you add is not one dependency. It is a tree. When your RAG framework has hundreds of transitive dependencies and executes in the same process that holds your database credentials, the framework's security posture is your security posture.

Shadow AI: the surface you did not approve at all

Everything above covers the stack you chose. There is also the stack you did not: employees pasting source code, contracts, and customer records into whatever consumer chatbot they prefer. Bans do not work; the incentive to use these tools is too strong. The pattern that does work is replacement: give people one sanctioned surface with access to the models they want, behind SSO, governed and logged. You cannot firewall your way out of shadow AI. You can only out-compete it.

How do you actually reduce an AI attack surface?

Vendor-neutral answer first. Four principles, in priority order:

Minimize copies. Every system that holds prompts, embeddings, or memories is a copy. Fewer systems holding data beats more systems holding it well.
Broker all egress. Model traffic should leave through one controlled point where you can enforce policy per workload, not through N SDKs each dialing their own provider.
Consolidate the control plane, keep the keys. One place to rotate credentials, one audit trail answering "which model saw which data," with keys that remain yours (BYOK).
Make deletion an API call. If you cannot demonstrate deletion across every copy programmatically, your retention policy is a hope, not a control. You can implement all four yourself with enough glue code and discipline. Some teams do.

Where we stand (the disclosed bias)

Backboard is the consolidated version of that stack: routing across 17,000+ models, memory (ranked first on the LoCoMo and LongMemEval benchmarks, receipts on GitHub), embeddings, agentic hybrid RAG, and stateful threads behind one API and one key. Enterprise deployments run as a signed container inside the customer's own cloud, behind the IAM, SOC, and audit infrastructure that already passed review. Egress to external models is opt-in per workload, never standing. Memory has full CRUD endpoints, so export and deletion are API calls you can put in a runbook.

The fair objection: "you just built a single point of compromise." Three answers. Consolidating the control plane is not consolidating the keys; BYOK means compromise of the platform does not hand over your credentials. In the enterprise deployment the one door is a door you already own, inside your own perimeter. And memory is exportable and the platform is model-agnostic, so consolidation does not mean lock-in.

You will also notice no certification badges in this post. That is deliberate. We publish no certification we have not earned. Ask for the architecture, not the badge.

FAQ

Are embeddings personal data under GDPR?
Treat them as if they are. Embedding inversion research shows vectors derived from personal data can be reconstructed into close approximations of the source text, which makes "anonymized because it is numeric" a hard position to defend.

Is a unified AI platform a single point of failure?
It concentrates the control plane, which is exactly what makes it auditable. The security question is where keys live and where data can go. With BYOK and deployment inside your own perimeter, the failure domain is one you already operate, instead of eight you do not.

How many vendors are in a typical production LLM stack?
Six to nine: gateway, vector database, memory, RAG framework, embeddings, orchestration, observability, and two to three model providers. Count yours by listing every AI-related line item in billing and every AI-related SDK in your lockfiles. The two lists rarely match, and the gap is unreviewed surface.

What is the fastest single improvement?
Egress. Route all model traffic through one brokered, policy-enforced path and turn off direct provider access from application code. It is the change with the highest ratio of risk removed to engineering effort.

Does consolidation hurt model choice?
It should not, and that is a test to apply to any platform: consolidation of the data plane and control plane is valuable, consolidation that locks you to one model is a different product. Insist on model-agnostic routing and exportable state.

Anyone ever go back to their notetaker outputs and summaries?

Jonathan Murray — Fri, 21 Aug 2026 21:12:59 +0000

Nash Is Now Open Source; Meet Your New AI Workspace

Jonathan Murray — Fri, 21 Aug 2026 14:48:59 +0000

Nash Is Now Open Source 🎉

Big news: Nash is now open source.

You can find the repo here 👉 github.com/Backboard-io/nash

Nash started as an internal tool to make AI actually usable day-to-day — not a chat window bolted onto a model, but a real workspace. As of this release, it's open for anyone to use, fork, self-host, contribute to, or build on top of.

Alongside open-sourcing the project, we're also shipping a major release that pulls Nash's features into a single, connected experience. Here's what's new.

🎨 Image Generation, End-to-End

Image generation is now integrated end-to-end inside Nash.

Go from a rough idea → a first draft → iterations → a finished image without ever leaving the conversation. No context switching, no copying prompts between tools, no losing your thread.

Create, iterate, and keep working from the same experience.

📄 Reliable Document Creation

Whether you're turning an idea into a draft or generating a document as part of a larger workflow, Nash can now carry that process through from request to finished output.

Ask for a PDF, a spec doc, a Word file, a code file — and Nash treats it as a first-class output rather than a side effect of chat.

🔌 MCP Connectivity (Including Google MCP)

This release adds support for MCP (Model Context Protocol), including Google MCP, expanding what Nash can access and work with as part of your workflows.

Instead of keeping your AI isolated from the rest of your work, MCP gives Nash a way to connect with the systems around it — the tools, data, and services you already use every day.

If you've been waiting for an assistant that actually plugs into your stack instead of living beside it, this is the update for you. And since Nash is open source, you can wire up your own MCP servers, extend the integrations, or contribute new ones back to the project.

🖼️ A Library for Your Generated Images

Ever generated the perfect image and then lost it three conversations later? Same.

Nash now has a Library where your generated images are stored automatically, giving you one place to browse and access previous creations directly. Nothing to configure — it just works.

The Bigger Picture

This release is about making Nash feel less like a collection of AI features and more like one connected workspace:

End-to-end image and document generation means outputs feel first-class.
MCP means Nash lives inside your ecosystem, not next to it.
The Library means your work sticks around.

And now, all of it is open source. You can read the code, self-host it, extend it, or ship a fork of your own.

Get Involved

We'd love your help shaping where Nash goes next:

⭐ Star the repo: github.com/Backboard-io/nash
🐛 Open issues for bugs, ideas, or feature requests
🔧 Send PRs — integrations, MCP servers, UI improvements, all welcome
💬 Try it live at hellonash.ai

Ask for something. Create it. Connect the tools you already use. Come back to your work later.

Say hello to your new AI workspace. 👋

Benchmarks Don't Build Great Products. Engineers Do.

Jonathan Murray — Fri, 21 Aug 2026 14:26:26 +0000

The AI industry has a benchmark problem. Not because we have too many benchmarks. Because too many companies treat them like trophies instead of tools.

A benchmark's primary job is to make your product better. Publishing the score is secondary. The test I use: a benchmark should challenge your engineers before it impresses your marketing team.

If it isn't making your product better, it probably isn't serving its most important purpose.

Why does the AI industry have a benchmark problem?

If you've followed AI over the past year, you've seen an endless stream of benchmark announcements. Every week another model reaches the top of another leaderboard. Every release claims a new state of the art. Every company seems to have a chart proving they're the best.

It's easy to become cynical.

The problem isn't that benchmarks exist. Benchmarks are one of the most valuable engineering practices we have.

The problem is that we've started confusing the measurement with the mission.

At Backboard, we don't benchmark because we believe benchmarks are perfect. We benchmark because imperfect, transparent measurements are better than subjective claims.

That idea shapes how we think about engineering.

What are benchmarks actually for?

Benchmarks are feedback loops. That's the whole point.

A benchmark tells us where we're strong, where we're weak, and whether the changes we made actually improved something meaningful. Sometimes an optimization delivers exactly what we hoped for. Other times it exposes a regression we never expected.

Without objective evaluation, it's remarkably easy to convince yourself your product is getting better simply because you've spent weeks working on it.

Benchmarks have a way of keeping engineers honest.

They also create a common language. Our customers shouldn't have to rely solely on our opinion of our own products. Public evaluations give everyone a shared point of reference. No benchmark captures every aspect of an AI system, but transparent measurements let different products be compared using the same criteria.

That's healthier than a world where every company simply declares itself the best.

How do benchmarks get abused?

Like any measurement, they can be gamed. Four ways, all common:

Optimizing specifically for the benchmark rather than the capability it's supposed to measure
Leaking evaluation data into training
Cherry-picking configurations until one produces a good number
Publishing only the results that make for impressive headlines None of those things improve the product.

They improve the marketing.

What's the difference between benchmarking what you built and building for benchmarks?

This is the distinction that matters, and the two can produce similar-looking leaderboard results while being fundamentally different engineering philosophies.

	Building for benchmarks	Benchmarking what you built
Starts with	The test	The customer
Goal	Maximize a score	Validate real improvement
Optimizes	The evaluation set	The capability
When the score improves	Ship the announcement	Check whether customers feel it
When the score drops	Find a better configuration	Find the regression
Failure mode	A number that never translates to value	A slower news cycle

Building for benchmarks starts with the test. The goal becomes maximizing a score, even if that improvement never translates into real world value.

Benchmarking what you've built starts with the customer. You solve real problems first, then use independent evaluations to validate you're moving in the right direction.

Build first. Benchmark second.

What does benchmark transparency actually look like?

Transparency matters as much as performance. Whenever possible we publish our methodology, open source our evaluation frameworks, and share the logs, configurations, and results so others can reproduce our findings.

If someone discovers we've made a mistake, that's not a failure of the process. It's evidence the process is working.

Science advances because results can be challenged. Engineering improves because assumptions are tested.

Here's what that means concretely.

In July 2026 we published an R-CLI result of 84.3% (75 of 89 tasks) on Terminal Bench 2.1, running Claude Opus 4.8 via Bedrock. That's above every published result we're aware of, including Codex CLI at 83.4% and Claude Code at 83.1%. On the same underlying model, R-CLI scored 5.4 points higher than Claude Code.

We also published 72% with GLM 5.2, an open source model. That number is less flattering and we published it anyway, because a company that only shows you its best configuration is showing you marketing.

We did not claim a ranking. Terminal Bench isn't currently accepting submissions, so "highest published result" is accurate and "number one on the leaderboard" would not be. That distinction is small, and getting it right is the entire job.

The per-task verifier logs are public: github.com/Backboard-io/Backboard-R-CLI-Terminal-Bench-2.1-Results

Our memory results, which lead LoCoMo and LongMemEval, are published the same way: github.com/backboard-io

We want people to inspect the work, reproduce it, challenge it, and improve on it. That creates far more confidence than posting a screenshot of a leaderboard ever could.

What don't benchmarks measure?

No benchmark will ever tell the entire story.

They don't measure customer trust. They don't measure usability. They don't capture every workflow or every edge case that matters to an enterprise.

Public benchmarks should always be complemented by real customer evaluations, production deployments, and continuous feedback.

Benchmarks are one input. Not the only input.

Is it wrong to benefit from benchmark reach?

No, and I'm happy to acknowledge it.

Benchmarks create reach. Strong benchmark performance helps people discover what we're building. It starts conversations with engineers, customers, investors, and partners who otherwise might never have found us.

There's nothing wrong with that.

What's important is the sequence:

Benchmark to build a better product
Be transparent about the methodology
Share the results
Learn from the feedback
Repeat Reach is the byproduct. Not the objective.

FAQ

What is the purpose of an AI benchmark?
To create a feedback loop that tells engineers whether a change actually improved the product, and to give customers a shared point of reference instead of a vendor's opinion of itself.

How can you tell if a company is gaming a benchmark?
Look for published methodology, open evaluation frameworks, raw logs, and per-task results. Look for whether they publish their weaker configurations too. A single screenshot of a score is not evidence.

What's the difference between building for benchmarks and benchmarking what you built?
Building for benchmarks starts with the test and optimizes the score. Benchmarking what you built starts with the customer and uses evaluations to validate that real improvements happened.

Are benchmark scores enough to evaluate an AI system?
No. Benchmarks don't measure trust, usability, or enterprise edge cases. They should be one input alongside customer evaluations and production feedback.

Why publish evaluation logs and not just scores?
Because a score you can't reproduce is a claim, not a result. Logs let other people verify the work, and being corrected is how the process is supposed to function.

What has Backboard published?
R-CLI at 84.3% on Terminal Bench 2.1 with Claude Opus 4.8, and 72% with the open source GLM 5.2, both with per-task verifier logs on GitHub. Backboard's memory system leads LoCoMo and LongMemEval, published the same way.

The leaderboard is a byproduct, not the objective

We hope our products perform well on public evaluations. Of course we do.

But that's never been the goal.

The goal is to build software that genuinely helps people solve difficult problems. If we ever stop learning from benchmarks and start treating them as trophies, we'll have missed the point entirely.

Because in the end, benchmarks don't build great products.

Engineers do.

How to Let OpenRouter Pick Your Model, and Choose Who Serves It

Jonathan Murray — Fri, 21 Aug 2026 14:06:54 +0000

Backboard now supports OpenRouter automatic model selection (openrouter/auto) and per request provider selection. Here is how both work, with JSON examples for the Backboard API.

Short answer

Backboard now gives you two independent routing controls for OpenRouter requests:

Automatic model selection. Set llm_provider to openrouter and model_name to openrouter/auto. OpenRouter classifies the prompt and picks the model. You pay the standard rate of the model it chooses, with no additional router fee.
Provider selection. Pin a specific model to a specific upstream provider with the openrouter.providers array, and keep allow_fallbacks on so the request still completes when that provider is unavailable. Both are live now in the Backboard API and in the Python and TypeScript SDKs from v1.5.16 onward.

Why model routing is now two decisions, not one

Most LLM routing writeups treat "which model" as the whole question. It is not.

OpenRouter aggregates 17,000+ models, and many of those models are served by more than one upstream provider. The same model weights can sit behind different pricing, different throughput, different context handling, and different uptime depending on who is running them. So there are two decisions in every request:

Which model should answer this prompt?
Who should run that model? Until this release you answered the first question and inherited an answer to the second. Now you can set both, or delegate both, per request.

How do you let OpenRouter choose the model automatically?

Set the model name to openrouter/auto. OpenRouter classifies the incoming prompt and routes it to a model it judges appropriate for that class of work.

{
  "llm_provider": "openrouter",
  "model_name": "openrouter/auto",
  "openrouter": {
    "cost_tier": "low"
  }
}

This is useful when your traffic is mixed. A support inbox that receives one line acknowledgements and multi page technical escalations does not need the same model for both. Automatic selection sizes the model to the prompt instead of forcing you to build that classifier yourself.

Does openrouter/auto cost extra?

No. You pay the standard rate of whichever model OpenRouter selects. There is no additional router fee layered on top.

How do you constrain automatic model selection?

Automatic does not have to mean unbounded. Three fields inside the openrouter object narrow the candidate set before selection happens.

Field	What it does	Example
`allowed_models`	Restricts selection to a list or pattern of models	`["anthropic/*"]`
`excluded_models`	Removes specific models from consideration	`["some-vendor/experimental-model"]`
`cost_tier`	Caps how expensive a model the router may reach for, from `low` to `max`	`"low"`

allowed_models: ["anthropic/*"] limits automatic selection to that vendor family. That pattern matters for teams with a procurement, residency, or vendor approval constraint. You get automatic selection inside a boundary you defined, rather than automatic selection across everything.

cost_tier is the blunt lever. Set it to low for high volume, low stakes traffic. Raise it for work where an extra few cents per call is irrelevant next to the cost of a wrong answer.

How do you choose which provider serves a model?

Pass a providers array inside the openrouter object. The request is routed to that provider for the model you named.

{
  "llm_provider": "openrouter",
  "model_name": "moonshotai/kimi-k3",
  "openrouter": {
    "providers": ["together"],
    "allow_fallbacks": true
  }
}

Here the model is fixed and the provider is fixed. You chose moonshotai/kimi-k3, and you chose who runs it.

This matters when you have benchmarked providers against each other and found a real difference, when one provider's pricing for a given model is materially better, or when you have an existing commercial relationship with one of them.

What happens when your chosen provider is unavailable?

That is what allow_fallbacks is for. With allow_fallbacks: true, the request can fall back to another provider serving the same model when your selected provider is unavailable. Your preference is honored when it can be, and the request still completes when it cannot.

Set it to false when the provider choice is a hard requirement rather than a preference, and you would rather see the request fail than silently run somewhere else.

Which option should you use?

Your situation	Use
Mixed prompt complexity, no strong model preference	`openrouter/auto` with a `cost_tier`
Vendor family is constrained but model choice is not	`openrouter/auto` with `allowed_models`
You know exactly which model you want	Pin `model_name`, leave provider unset
You know the model and the provider you want	Pin `model_name` plus `openrouter.providers`
Provider choice is a hard requirement	`providers` with `allow_fallbacks: false`

These are per request settings, not account level settings. Different endpoints in the same application can make different choices.

How do you know which model actually ran?

The response tells you. Backboard returns the provider and model that handled the request alongside token counts, so automatic selection does not become a visibility gap. You can log what ran, attribute cost to it, and audit routing behavior after the fact.

That is the practical objection to automatic routing, and it is the reason the response carries the answer rather than leaving you to infer it.

Where can you see which providers serve a given model?

The Backboard Model Library now surfaces provider options for OpenRouter models, alongside pricing, context limits, uptime, and other model information.

Browse it here: https://app.backboard.io/dashboard/model-library

Check the library before you hardcode a provider name. Provider availability for a given model changes.

What do you need to use this?

A Backboard API key from https://app.backboard.io
llm_provider set to openrouter
Backboard API, or the Python or TypeScript SDK at v1.5.16 or later Backboard's free tier includes $5 in memory credits and requires no credit card. Inference is billed separately.

Full parameter reference: https://docs.backboard.io/concepts/messages

FAQ

What is openrouter/auto?
It is a model identifier that hands model selection to OpenRouter. Instead of naming a specific model, you name openrouter/auto, and OpenRouter classifies the prompt and selects a model for it.

Does automatic model selection add a fee?
No. You pay the standard rate of the model that gets selected.

Can I limit automatic selection to certain models?
Yes. Use allowed_models to restrict the candidate set, excluded_models to remove specific models, and cost_tier to cap price, on a scale from low to max.

Can I choose which provider serves a model on OpenRouter?
Yes. Pass a providers array inside the openrouter object in your Backboard request body.

Does provider selection work with automatic model selection?
Provider selection applies to a model you have named. If you delegate model choice to openrouter/auto, you are also delegating the provider that serves it.

What does allow_fallbacks do?
With allow_fallbacks: true, a request can move to another provider serving the same model when your selected provider is unavailable. With it off, the provider choice is strict.

Which SDK versions support this?
Python and TypeScript SDK v1.5.16 and later, plus the Backboard API directly.

How do I find out which model handled my request?
The API response reports the provider and model that ran, so automatic selection stays auditable.

Summary

model_name: "openrouter/auto" delegates model choice to OpenRouter at no extra fee.
allowed_models, excluded_models, and cost_tier keep that delegation inside boundaries you set.
openrouter.providers picks who serves a named model, and allow_fallbacks decides whether that pick is a preference or a rule.
The response reports what actually ran.
Available now in the Backboard API and in the Python and TypeScript SDKs from v1.5.16. Choose the model. Choose who runs it. Backboard handles the rest.

Coding agents got boring the moment we built a really good one.

Jonathan Murray — Mon, 17 Aug 2026 20:12:46 +0000

MESSAGE FROM BACKBOARD.IO Co-Founder Rob Imbeault:

Coding agents got boring the moment we built a really good one.

That sounds dismissive. I’m ok with that.

Cursor is worth tens of billions. Coding agents are attracting enormous amounts of capital. The category is clearly valuable.

But building R-CLI changed how I think about where the value actually sits.

We built one of the best coding harnesses in the world in a matter of months. On Terminal-Bench 2.1, it performs at the frontier.

Then we open sourced the entire thing.

Not because it wasn't valuable.

Because building it convinced us that the value is moving somewhere else.

Frontier agent performance is becoming accessible much faster than I expected. Small teams can now build systems that compete with products coming out of organizations with vastly more capital, people and compute.

That should make anyone building in this space ask an uncomfortable question:

If a small team can build a frontier coding agent in a few months, how durable is the agent itself as a technical moat?

I don't think the answer is very.

There will still be enormous companies built around coding agents.
Distribution is hard. Product is hard. Workflow ownership is hard.
Brand is hard. Building a great company is very hard.

But the agent?

Increasingly, I think that's becoming the easy part.

The problems that still feel genuinely hard are underneath it.

Making models dramatically cheaper to run.

Giving them persistent organizational context.

Improving them with proprietary data.

Running them privately, locally and securely.

Making open models perform closer to frontier models.

Controlling how intelligence moves across an organization.

Those are the problems we want to work on.

We didn't build R-CLI because we want to win the coding agent market.

We built it and accidentally discovered why we don't.

So rather than build the 1,001st proprietary coding agent and protect the harness like it is the moat, we open sourced ours.

Because our bet is that the next great AI infrastructure companies will not own one agent.

They will make thousands of agents better.

Backboard-io / Backboard-R-CLI

Backboard R-CLI

Backboard R-CLI is an AI coding agent that runs in your terminal. Open it in a project and describe what you want to accomplish. It can inspect and edit files run commands, search the web, use MCP servers and skills, and keep its work reviewable through permissions and checkpoints.

This directory contains the TypeScript implementation of the CLI. End users should install the precompiled backboard binary. Contributors can run or compile it from source with Bun.

Documentation

This README covers installation, authentication, first use, and development. The maintained product guides contain the complete feature reference:

Install the CLI

The production installer downloads a native binary for your operating system. You do not need Bun, Node.js, or Python to use an installed binary.

macOS and Linux

curl

…

View on GitHub

We Open Sourced R-CLI, the Coding Harness Above Every Published Terminal Bench 2.1 Result

Jonathan Murray — Thu, 13 Aug 2026 14:12:06 +0000

R-CLI is now open source. MIT license. The whole harness.

Repo: https://github.com/Backboard-io/Backboard-R-CLI

First, the ask

We are giving away the harness behind the highest scores ever recorded on Terminal Bench 2.1. Free, MIT, running on your own keys in about two minutes. The receipts are below, with public verifier logs for every claim.

The entire price:

Star the repo. Stars are how open source projects get found. Thirty seconds.
Share it with one developer who pays too much for AI coding. That is it. This is a win for the community and for open source, and we want it loud.

Three ways to run it

One command on macOS or Linux:

curl -fsSL https://app.backboard.io/api/cli | sh

Windows PowerShell:

irm https://app.backboard.io/api/cli/windows | iex

Or build from source, because now you can: clone the repo, bun install --frozen-lockfile, then bun run build for a native binary on your OS. Bun, Git, and ripgrep are the only prerequisites.

Here is why this matters, and why we gave it away.

The thesis: the harness is the dominant variable

The industry story is that coding performance lives in the model. Pick the biggest frontier model, pay the bill, get the results.

Our runs say otherwise. Take the same model, change only the harness, and the score moves more than a model generation's worth.

R-CLI is built on the RLM, a recursive coding engine. The parent agent delegates work off-model to child agents that run in bounded contexts. Each child gets a sharp, small slice of the problem instead of one giant context window accumulating noise. Work comes back up, context gets compressed, and the loop continues. That structure is why the same model scores higher inside R-CLI, and why it does it with 10 to 30 percent fewer tokens.

The numbers, with receipts

Two published runs from our technical reports. Per-task verifier logs are public at https://github.com/Backboard-io/Backboard-R-CLI-Terminal-Bench-2.1-Results

Same model, better harness. Claude Opus 4.8, full Terminal Bench 2.1 submission shape: 89 tasks, 5 attempts each, 445 trials, no concessions. R-CLI scored 85.4 percent. Claude Code's published score with the same model is 78.9 percent. That is +5.14 points with exactly one variable changed. The result sits above all 17 published entries. It has not yet been submitted for judge review, so treat it as a published run with public logs, not a leaderboard entry.

No frontier model in the run. Moonshot AI's Kimi K3 via OpenRouter, with DeepSeek V4 Flash handling subagent work. Single attempt per task: 91.0 percent (81 of 89). Total cost for the entire run: $28.72. About 32 cents per task. Caveat carried in the report itself: single-attempt runs have a roughly ±3 percent confidence interval and this is not a judged submission. Even at the bottom of that interval, an open-weight model in this harness is playing at the top of the benchmark for a rounding error of the usual cost.

One run shows the harness lifts a frontier model past its own tooling. The other shows the harness makes frontier-level coding nearly free. Same engine both times.

What you actually get

Terminal-native coding agent: inspects files, edits, runs commands, searches the web
The recursive engine: off-model delegation, bounded child contexts, context compression
Hundreds of pre-loaded skills, browsable with /skills, plus your own in .agents/skills/
Hundreds of MCP connections, managed with /mcp
Permission modes: manual, acceptEdits, bypass. You decide what it can touch
Checkpoints: undo and redo agent changes, reviewable diffs
Session logging, one-shot mode, JSON output for scripting, hooks via /hooks ## Any model. Your keys.

R-CLI is model-agnostic, with a path to hundreds of models. Bring your own key for OpenAI, OpenRouter, Anthropic, or Grok, or sign in with Backboard. Switch with /model. When a better or cheaper model ships next month, you change one setting. No re-platforming, no new tool.

The part that matters most: run it where your code lives

Every closed coding tool ships your source to a vendor API. That is the deal, whether the marketing says so or not.

Pair R-CLI with an open model you host yourself and that deal is off. Frontier-level coding where your code never leaves your infrastructure. The Kimi K3 run is the proof that this path is not a performance downgrade.

And now the harness itself is MIT. You can read it, audit it, fork it, and run it inside your own walls. Nobody can policy-block it, deprecate it, or reprice it out from under you.

Why open source it

Because the thesis demands it. If we claim the harness is the dominant variable, the harness should be inspectable. Benchmarks with public verifier logs were half of that. Open code is the other half.

And because ownership is the whole point of what we build. Own the model. Own the harness. Run both in your own cloud.

The ask, one more time

We gave away the harness. You star and share it. That is the trade.

https://github.com/Backboard-io/Backboard-R-CLI

Then install it, break it, open issues, send PRs:

curl -fsSL https://app.backboard.io/api/cli | sh

Docs: https://docs.backboard.io

We hit 99.95% on the LoCoMo memory benchmark. Here's the catch, and why it still matters.

Jonathan Murray — Wed, 12 Aug 2026 02:12:30 +0000

Our CEO Rob Imbeault published a piece on LinkedIn this week about a result our team posted: 99.95% on LoCoMo, the most cited benchmark for long-term conversational AI memory. His version is written for executives and CISOs. This is the developer version, because the interesting part isn't the number. It's how we got it, and what it says about where memory architecture is heading.

(Credit where it's due: the ideas here are from Rob's original article. The translation to dev-speak is mine.)

The catch, stated upfront

We didn't get 99.95% with retrieval. We got it by post-training the memory directly into the model weights, and we trained on the same conversation set the benchmark tests. On purpose.

So no, this doesn't prove our model is "better at memory" than yours. It was never meant to. The experiment measures the ceiling of parametric memory: when you teach a model a corpus of conversations, how well can it recall and reason over them with zero retrieval machinery? The answer turns out to be: near-perfectly.

That result matters because the technique behind it just became cheap. And that changes the architectural conversation for everyone building on LLMs.

The problem: every LLM app has goldfish memory

You know the drill. The model forgets everything when the conversation ends, so we all built the same workaround: chop conversations into chunks, embed them, index them, and at query time search the store and staple the relevant chunks to the front of the prompt.

RAG. It works. Almost every "AI with memory" product ships it. But it has three structural costs that show up at scale:

The token tax. Every query re-sends the same context. Customer history, policy docs, prior decisions, reloaded and re-billed, forever.
Shared-store leakage. Isolation in a shared vector store is a config property: a filter, a tag, a permission check. Config fails. The industry's incident list of one tenant's memory surfacing in another tenant's session keeps growing.
Silent retrieval failures. Grab the wrong chunk and the model confidently answers the wrong question. No stack trace, no error, just a wrong answer. ## The other approach: teach it into the model

Post-training used to be expensive and exotic. In 2026 it isn't. LoRA makes the teaching efficient. Quantization (our BBQuant work lives here) shrinks the result without meaningful quality loss. Together they've turned post-training from a research project into something a small team can run as a routine operation, on modest hardware.

Teach the corpus into the weights and the properties flip:

No token tax. The memory lives in the model. You pay a one-time teaching cost, then queries are just queries.
Isolation by construction. Each tenant gets their own weights. One tenant's model physically cannot see another tenant's data. There's no shared store to misconfigure.
It's a file. A model with its memory baked in runs on a laptop, a factory-floor box, or an air-gapped server. Nothing phones home. The trade-offs are just as real, and you should weigh them honestly. Updating is slower than a database write. Deleting a specific fact is genuinely hard, which matters under GDPR and PIPEDA right-to-be-forgotten regimes. And parametric memory is better at recalling what it was taught than reasoning about brand-new conversations it has never seen.

Measuring it properly: LoCoMo-Δ

Because we trained on the test material, recall and generalization are tangled together in our 99.95%. The benchmark as it stands can't separate them. So alongside the result we're proposing LoCoMo-Δ, an extension that holds conversations back from training, letting the field measure recall and generalization independently.

Benchmarks only mean something if they measure the right thing. If you work on memory or eval design, we'd like your eyes on the protocol.

The math for repeated workloads

Here's the part I find most interesting as an engineering decision. For any corpus that gets queried repeatedly, there's a crossover point where one-time teaching beats per-query retrieval. Depending on corpus size and query volume, that crossover lands somewhere between hundreds and thousands of queries.

Support bots, internal knowledge bases, sales enablement, compliance Q&A: most of these clear that threshold in about a week. The cost curve everyone assumes is fixed ("memory = tokens forever") is only fixed because retrieval was the only tool on the shelf.

The honest architecture: both

This isn't "RAG is dead." The right answer is deliberately splitting the job:

Parametric memory for stable, high-value, per-tenant knowledge: playbooks, policies, product knowledge, recurring customer context. The stuff a new hire eventually internalizes and stops looking up.
External memory for volatile, user-owned, or regulated data: recent transactions, preferences, anything that must be edited, deleted, or audited on demand. Most systems today use retrieval for both jobs. Not because anyone chose that, but because nobody offered the choice.

What we're releasing

This is the part that should matter to you even if you never touch our platform:

The post-training recipe we used, reproducible on modest hardware
A BBQuant-compressed checkpoint small enough to run on a high-end laptop
The LoCoMo-Δ protocol proposal for the research community Watch github.com/backboard-io for the drops.

The 99.95% is the headline. The real story is that the tooling to get there quietly became accessible to almost everyone. If you're building anything with long-lived context, the memory architecture decision is now actually a decision. Better to make it on purpose than inherit it from whichever vendor you started with.

We're a small Canadian team at Backboard building model-agnostic AI infrastructure, and we like showing our work. If you disagree with any of the above, the comments are open. That's what they're for.

Original article: When Your AI Remembers: A New Way to Think About Memory in Enterprise AI by Rob Imbeault, CEO of Backboard.

Backboard CLI: 85.4% on Terminal-Bench 2.1, submitted to the official leaderboard

Jonathan Murray — Wed, 12 Aug 2026 02:02:37 +0000

The result

We submitted the Backboard CLI to the official Terminal-Bench 2.1 leaderboard this week. The numbers:

Accuracy: 85.4% ± 0.8%
Pass@5: 0.888
Coverage: 89 tasks × 5 attempts = 445 trials, all included
Model: Claude Opus 4.8 via AWS Bedrock
Total cost of the full run: $280.72 The submission is above every published entry on the leaderboard. The current top published results are Claude Code with Fable 5 at 83.8% ± 1.2% and Codex with GPT-5.5 at 83.1% ± 1.1%. Our PR is open and pending review, and every trial log is public.

Why this matters

Terminal-Bench 2.1 measures what developers actually do: real multi-step tasks in a real terminal. Build failures, dependency messes, debugging under constraints. It's one of the most credible agentic coding benchmarks running.

The leaderboard's top entries are agents built by the same labs that built the underlying models. Claude Code is Anthropic's agent on Anthropic's model. Codex is OpenAI's agent on OpenAI's model.

The Backboard CLI outscored both. On a model we don't make.

That's the point. The harness does the work. How an agent decomposes tasks, delegates to bounded child contexts, and manages what enters the context window determines how much capability you get out of a model. Our CLI is built on a recursive coding engine designed around exactly that, which means performance isn't chained to any single frontier model. Same harness, different model, and it still performs. We've run the same harness on GLM 5.2, a fully open-source model, and scored 72% on this benchmark. That matters if you care about cost, or about running coding agents on infrastructure you control.

How we ran it

No cherry-picking, no re-minting, no overrides:

Five sequential jobs under one identical agent config
Pinned dataset (SHA-256 verified), default settings
All 445 trials included in the reported number
Errored trials counted as zero reward
Full logs uploaded and public Benchmarks only mean something if you can check them. Check ours: PR #200.

We're open-sourcing the CLI

The Backboard CLI is going open source. Soon.

We build model-agnostic infrastructure. The whole thesis is that you shouldn't be locked into one lab, one model, or one vendor's agent. Open-sourcing the CLI is that thesis in practice: take the harness, point it at the model you want, run it where you want.

Watch this space, or follow us for the release announcement.

Try it today

You don't have to wait for the repo. The CLI is available now at backboard.io/cli.

Backboard is a Canadian company building full-stack, model-agnostic AI infrastructure: memory ranked #1 on LoCoMo and LongMemEval, routing across 17,000+ models, agentic hybrid RAG, and stateful threads behind one API key. The CLI is one surface of that platform.

Use any model. Keep your options open. Ship faster.

Sources: Terminal-Bench 2.1 leaderboard | Leaderboard submission PR #200

Our small team just outscored Claude Code and Codex on Terminal-Bench

Jonathan Murray — Wed, 12 Aug 2026 02:00:26 +0000

I want to be upfront about something: I didn't build this. Our engineering team did. But I get to write about it, and I've been waiting a while to write this one.

This week we submitted the Backboard CLI to the official Terminal-Bench 2.1 leaderboard. The score: 85.4% ± 0.8%, with a pass@5 of 0.888, running Claude Opus 4.8 via Bedrock.

For context, the top published entries on the leaderboard right now are Claude Code with Fable 5 at 83.8% and Codex with GPT-5.5 at 83.1%. Those are Anthropic's and OpenAI's own coding agents. Built by the labs that built the models.

Our submission is above every published result. It's pending review on the leaderboard now, and you can go look at every trial yourself.

Who we are

Backboard is a fledgling startup out of Nepean, Ontario. Not San Francisco. Not a lab with a frontier model and a billion dollars of compute. A Canadian team building AI infrastructure, and a CLI coding agent as one piece of it.

Terminal-Bench is one of the harder agentic benchmarks out there: 89 real terminal tasks, the kind of messy multi-step work that actually looks like a developer's day. The leaderboard is dominated by the biggest names in AI running their own models through their own agents.

So when our team's harness put up a higher score on a model we don't even make, that told me something I've believed for a while: the harness matters as much as the model. How an agent plans, delegates, and manages context decides how much capability you actually get out of any model you point it at.

How the run went down

The team didn't cherry-pick. Five sequential jobs, one identical config, all 89 tasks, 5 attempts each. That's 445 trials, every one of them included. Errored trials counted as zero. Pinned dataset, default settings, no overrides, all logs public.

Total cost of the entire run: $280.72.

I love that number. Frontier-level agentic coding performance, fully benchmarked, for less than a conference ticket.

We're open-sourcing it

This is the part I'm most excited about. We're going to open source the CLI. Soon.

A fledgling startup doesn't win by hoarding. We win when developers actually use the thing, kick the tires, break it, and tell us what's wrong with it. The benchmark score gets people to look. The tool has to earn the rest.

Try it

The CLI is available today at backboard.io/cli. It's model-agnostic, so you're not locked into any one provider.

If you try it, I'd love to hear what you think, good or bad. We're small enough that your feedback actually reaches the people who built it. Usually the same day.

And to the team: you know who you are. This one's yours.

Receipts: PR #200 on the Terminal-Bench 2.1 leaderboard repo | Terminal-Bench leaderboard

To big to lie, cheat, steal?

Jonathan Murray — Wed, 05 Aug 2026 00:25:59 +0000

Remember when Volkswagen cheated and lied on their emissions testing for their clean diesel… That was so crazy in retrospect.

Imagine Anthropic and OpenAI were completely training on everyone’s data and just lying about it and promising not to in their Terms and Conditions?

I guess we’ll see.

DEV Community: Jonathan Murray

I Counted the Attack Vectors in Our AI Stack and Now I Can't Sleep

the fun part: what each one actually is

quick quiz you will fail

"embeddings are just numbers though"

deletion theater

meanwhile, your dependencies

and the stack you didn't even choose

so what do you actually do

the horse, as promised

the whole post in four lines

Every Layer of Your AI Stack Is an Attack Vector. Count Them.

TL;DR

What is an AI attack surface?

The stack you actually deployed

Why vector databases are the layer nobody reviews

The deletion problem: one prompt, four retention policies

The supply chain you inherited

Shadow AI: the surface you did not approve at all

How do you actually reduce an AI attack surface?

Where we stand (the disclosed bias)

FAQ

Further reading

Anyone ever go back to their notetaker outputs and summaries?

Nash Is Now Open Source; Meet Your New AI Workspace

Nash Is Now Open Source 🎉

🎨 Image Generation, End-to-End

📄 Reliable Document Creation

🔌 MCP Connectivity (Including Google MCP)

🖼️ A Library for Your Generated Images

The Bigger Picture

Get Involved

Benchmarks Don't Build Great Products. Engineers Do.

Why does the AI industry have a benchmark problem?

What are benchmarks actually for?

How do benchmarks get abused?

What's the difference between benchmarking what you built and building for benchmarks?

What does benchmark transparency actually look like?

What don't benchmarks measure?

Is it wrong to benefit from benchmark reach?

FAQ

The leaderboard is a byproduct, not the objective

How to Let OpenRouter Pick Your Model, and Choose Who Serves It

Short answer

Why model routing is now two decisions, not one

How do you let OpenRouter choose the model automatically?

Does openrouter/auto cost extra?

How do you constrain automatic model selection?

How do you choose which provider serves a model?

What happens when your chosen provider is unavailable?

Which option should you use?

How do you know which model actually ran?

Where can you see which providers serve a given model?

What do you need to use this?

FAQ

Summary

Coding agents got boring the moment we built a really good one.

Backboard-io / Backboard-R-CLI

Backboard R-CLI

Documentation

Install the CLI

macOS and Linux

We Open Sourced R-CLI, the Coding Harness Above Every Published Terminal Bench 2.1 Result

First, the ask

Three ways to run it

The thesis: the harness is the dominant variable

The numbers, with receipts

What you actually get

The part that matters most: run it where your code lives

Why open source it

The ask, one more time

We hit 99.95% on the LoCoMo memory benchmark. Here's the catch, and why it still matters.

The catch, stated upfront

The problem: every LLM app has goldfish memory

Measuring it properly: LoCoMo-Δ

The math for repeated workloads

The honest architecture: both

What we're releasing

Backboard CLI: 85.4% on Terminal-Bench 2.1, submitted to the official leaderboard

The result