Missing Is Not False

A while back I wrote about what breaks when you turn government open data into something machines can trust — the quiet ways a publisher reshapes what it sends you, so the same question comes back a different shape depending on how (or when) you ask.

Then a reader, Vinh (@vinhnguyenthanhdn), left a comment sharper than my whole post. He didn't argue. He handed me a cleaner example — hiding inside an API I use every day. This one. DEV's.

Here's the catch.

Ask DEV for a single article the public way:

GET /api/articles/{id}

and the response has no published field at all. It has published_at, published_timestamp, a readable date — but no boolean that says "this is published."

Now ask the owner endpoint for your own posts:

GET /api/articles/me/all

and every article carries a published boolean.

Same article. Two representations. In one the field exists; in the other it was never there. I checked before I believed it — he's exactly right.

Why does that matter? Because of what your code does next. Almost every parser reaches for the field like this:

is_live = article.get("published")   # -> None on the public response

On the public response published is missing, so you get None, which is falsy, so your pipeline quietly decides the article is not published — when it plainly is. The request succeeded. The status was 200. Nothing errored. You just silently recorded the opposite of the truth.

And here's the part Vinh made me see clearly: "the publisher stopped sending this field" and "the publisher sent the field as false" are different events. Ordinary deserialization flattens both into the same value. One means draft. The other means I wasn't looking at a representation that carries this fact at all. Collapsing them isn't a rounding error — it throws away the one thing you needed to know: whether you actually have an answer, or just an absence.

That's not a bug to file against DEV. It's the shape of the whole problem I care about. Public data sources reshape what they send — across endpoints, across time — and the danger was never the reshape. It's that naive code turns "I don't know" into a confident "no."

In the original post I wrote the rule we run CivicDataForge on: an API should not manufacture certainty because a consumer wants a Boolean. Vinh read that and went one further. He didn't just report the bug — he handed me the fix for the whole class: treat absence and null as a monitored dimension of their own, and record the observed key set per endpoint next to the record hash, so "the publisher stopped sending this field" stays diffable from "the publisher sent it empty."

That sentence changed the company. We carried it into CivicDataForge's evidence envelope as a shape receipt on the source response — the keys we observed, the keys we expected but didn't get, which came back explicitly null, which came back explicitly false, and a fingerprint of the whole shape. It's live in our first services today, and we're rolling it out across the rest of the catalog, so field presence becomes evidence in its own right — not a parsing detail the deserializer swallows. A reader's comment became a schema. That's the best trade I've made all month.

And because the example deserved to be honored in its own language, we also built the small thing: a free DEV article checker that runs on exactly those rules. It now lives in the CivicDataForge catalog beside the permit and registry evidence tools — the side quest that proved the main quest. Nothing to sign up for.

You give it a link to a DEV article. It answers with one of three states, never fewer:

• PUBLISHED — seen live on the public endpoint (200). It tells you that's where the evidence came from.
• DRAFT — the owner representation says published: false. Different evidence, different door.
• UNKNOWN — the fact could not be observed. Not "no." Unknown. The state everyone else quietly throws away.

Every answer ships with a receipt — a hash of what was checked and when — so it's evidence, not vibes. And there's a version that runs on your own machine for checking your own drafts: your DEV key only ever talks to DEV. It never touches my server, never gets logged, never gets stored. I don't want your key. I want you to trust the answer.

Check any DEV article, free, no signup: https://civicdataforge.pages.dev/data/devto-publication-evidence

That's the whole idea, and it's Vinh's more than mine: missing is not false, and a system you can trust says so out loud. If it can't see something, it hands you an honest UNKNOWN instead of a confident wrong answer.

Thanks, Vinh. I asked which publishers silently reshape results — you answered, then showed me the gap in my own envelope. The company is better for it.