DEV Community: Avery Lin

A Free-Tier Docs Drafting Pipeline: What the Model May Commit and What You Own

Avery Lin — Thu, 27 Aug 2026 00:59:09 +0000

Documentation generation breaks at the ownership boundary more often than it breaks at the prompt, and the fix is structural rather than textual. A model can draft an API reference from an AST, but it cannot decide that a changed parameter breaks a compatibility promise. Free model access and a free server change the economics of regeneration, which makes the boundary rules the only part of the pipeline that matters.

The workflow below uses MonkeyCode's free model access and free server option to keep the marginal cost of a draft near zero. That cost profile lets the job run on every merge instead of once per release. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The artifact is a small Python script that extracts changed public symbols and checks them against your reference docs, and it produces a draft branch containing only the missing fragments.

Why Docs Rot Is a Cost Problem, Not a Quality Problem

Metered per-token generation gets skipped on boring commits, and boring commits are exactly where documentation rot begins. A one-line refactor rarely justifies a paid API call, so the paragraph about the refactored function stays stale until someone files an issue. A scheduled job with no per-run meter removes the skip decision entirely, because regeneration costs nothing at the margin.

The second reason is infrastructure friction, because a docs job that needs a dedicated VM lives on someone's laptop. A free server option removes that excuse, and the job becomes a cron entry that produces a draft branch while you sleep. None of this fixes prompt quality, and that is the point, because the prompt is the smallest part of a docs pipeline. The model's draft quality matters less than the rules that decide what happens to the draft.

Step 1: Write the Ownership Matrix Before the Prompt

The matrix is a contract between the model and the humans, and it must exist before the first draft is generated. Anything derivable from the diff belongs to the model, and anything that implies a promise belongs to a human.

Artifact	Model may draft	Human must own
API reference	Signatures, parameter names, return types	Parameter semantics, error behavior, edge cases
Changelog entry	Summaries from commit messages	Breaking-change classification, version impact
Migration note	Diff hunks and renamed identifiers	Compatibility promises, rollback plan
Architecture decision	Never	Rationale, alternatives, consequences
Security note	Never	Threat model, disclosure status, affected versions
Deprecation notice	Draft text only	Timeline, replacement guarantee, removal policy

The pattern is simple: the model owns what the diff proves, and the human owns what the diff implies. When a draft crosses a row in the owned column, the pipeline must stop and ask for a human, not silently commit.

Step 2: Schedule the Draft Job on the Free Server

The script below is the reproducible core of the pipeline, and it is deliberately small. It reads the diff between your base branch and HEAD, and parses changed Python files with the standard library. Each public symbol is then checked against your reference docs, and the missing ones are listed in a JSON report.

#!/usr/bin/env python3
"""docs_draft.py -- draft doc fragments for changed public symbols."""
import ast, json, os, subprocess, sys
from pathlib import Path

DOC_ROOT = Path("docs/reference")
BASE = os.environ.get("BASE_REF", "main")

def changed_files():
    out = subprocess.run(
        ["git", "diff", "--name-only", BASE, "HEAD"],
        capture_output=True, text=True, check=True,
    )
    return [p for p in out.stdout.splitlines() if p.endswith(".py")]

def public_symbols(path):
    tree = ast.parse(Path(path).read_text())
    found = []
    for node in tree.body:
        # top-level symbols only; extend for methods if needed
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and not node.name.startswith("_"):
            found.append(f"{path}::{node.name}")
        elif isinstance(node, ast.ClassDef) and not node.name.startswith("_"):
            found.append(f"{path}::{node.name}")
    return found

def covered(symbol):
    name = symbol.split("::")[-1]
    return any(name in p.read_text() for p in DOC_ROOT.rglob("*.md"))

def main():
    symbols = [s for f in changed_files() for s in public_symbols(f)]
    missing = [s for s in symbols if not covered(s)]
    report = {"changed_symbols": len(symbols), "undocumented": len(missing), "items": missing}
    print(json.dumps(report, indent=2))
    if missing and os.environ.get("DRAFT_DIR"):
        draft_dir = Path(os.environ["DRAFT_DIR"])
        draft_dir.mkdir(exist_ok=True)
        (draft_dir / "draft.md").write_text(
            "<!-- model draft: verify before commit -->\n"
            + "\n".join(f"## {s}\n\nTODO: one-paragraph description.\n" for s in missing)
        )
    return 1 if missing else 0

if __name__ == "__main__":
    sys.exit(main())

The script intentionally does not call a model, because the model call belongs in the orchestration layer. Point the job at MonkeyCode's free model access through any OpenAI-compatible client, and let the script decide which symbols need a draft. Schedule the job on the free server option as a cron entry or a scheduled workflow, and point the output at a draft branch. A typical schedule is nightly for a small repository, and the job should take under a minute for most diffs.

Step 3: Route Drafts to a Branch, Not to Main

The ownership matrix only works if the pipeline has a place to put drafts that is visibly not production. Follow these steps for every run:

Create or reset a branch named docs/draft from the current main.
Run the coverage script and write the missing fragments into docs/draft/draft.md.
Send the fragments to the model for a first-pass expansion, with the matrix as the system prompt.
Open a pull request labeled docs-draft, and never enable auto-merge on it.
Assign the PR to a human whose name appears in the owned column of the matrix.

The branch name and the label are the enforcement mechanism, because they make the draft's status visible to everyone. A draft that looks like a real PR will get reviewed like a real PR, which is exactly what you want.

Step 4: Measure Staleness With a Coverage Gate

The script returns exit code 1 when any changed symbol is undocumented, and that return code is the raw material for a trend. Run it on a schedule, store the JSON report, and track the undocumented count over time instead of treating each run as a one-off. A coverage gate that blocks merges will be disabled at the first Friday deadline, so keep this job as a report rather than a hard gate. Watch the delta between the previous run and the current run, because a rising delta means the matrix is being bypassed.

Limitations and Who Should Not Use This

Free-tier access generally implies rate limits and no service-level agreement, so this pipeline should never block a release. If the model is unavailable, the job should fail softly and leave the draft branch untouched, because stale docs are better than fabricated docs. Teams that maintain regulated documentation or external API contracts should not use this workflow at all. A wrong sentence about a compliance requirement is an incident, and a model draft is not a substitute for a human author in that domain. The same applies to security notes, where the cost of a confident error is measured in incidents, not in doc debt.

The Boundary Is the Pipeline

The free tier removes the cost excuse, and the server removes the infrastructure excuse, but neither removes the ownership decision. Write the matrix first, schedule the job second, and treat the model as a drafting intern who never merges their own work. If you already run a docs job, add the coverage script before you invest in a better prompt. The measurement will show you where the boundary is breaking, and that is the only signal worth optimizing.

A Free-Tier Docs Drafting Pipeline: Model Writes the First Pass, Humans Sign the Contract

Avery Lin — Wed, 26 Aug 2026 21:55:09 +0000

Documentation quality degrades for a structural reason, not a model reason: teams treat generation as one prompt instead of a pipeline with explicit ownership boundaries. A free-tier model and a free server can automate the mechanical first pass if you encode what the model may draft and what a human must sign. This article walks through a reproducible handoff pipeline that runs entirely on free resources, with a script, an ownership matrix, and a merge-time checklist.

Why generated docs rot

The common failure mode is not bad prose but unowned claims that nobody can trace back to a decision. A model that drafts architecture rationale, security notes, and onboarding paths is making decisions it cannot observe, and those decisions become documentation debt the moment they merge. API reference scaffolding, parameter tables, and changelog drafts are different because they derive directly from code that a human already reviewed. The pipeline below separates those two categories before any token is spent, which keeps the model in the drafting role and the human in the ownership role.

The ownership matrix

Before writing any code, define which artifacts the model may draft and which a human must own. The matrix below is the contract that both the prompt and the review checklist enforce.

Artifact	Model may draft	Human must own	Reason
API reference scaffolding	Yes	Review	Derives from signatures in the diff
Parameter tables	Yes	Review	Derives from function definitions
Changelog drafts	Yes	Edit	Needs release context and user impact
Getting-started prose	First pass	Rewrite	Needs real user paths and examples
Architecture rationale	No	Yes	Decision history is unobservable
Security and compliance notes	No	Yes	Liability and accuracy constraints
Runnable examples	No	Yes	Must execute against the real code

This matrix does two jobs at once: it constrains the drafting prompt, and it gives the human reviewer a concrete checklist for the merge gate. Teams that skip this step usually discover the problem in production, when a confident sentence about error handling turns out to be invented.

Step 1: Extract the diff context

The drafting stage should never ask the model to recall your codebase from memory. A small script reads the merged PR diff, collects changed function signatures and public exports, and writes a JSON context file that grounds the model in actual code.

#!/usr/bin/env python3
"""Extract public API changes from a diff for docs drafting."""
import json
import re
import subprocess
import sys

SIGNATURE_RE = re.compile(
    r"^(?:async\s+)?def\s+(\w+)\s*\(([^)]*)\)|^class\s+(\w+)",
    re.MULTILINE,
)

def changed_lines(base: str, head: str) -> list[str]:
    diff = subprocess.run(
        ["git", "diff", base, head, "--", "*.py"],
        capture_output=True, text=True, check=True,
    ).stdout
    return [ln for ln in diff.splitlines()
            if ln.startswith("+") and not ln.startswith("+++")]

def extract_signatures(lines: list[str]) -> list[dict]:
    found = []
    for ln in lines:
        for match in SIGNATURE_RE.finditer(ln):
            name, params, _ = match.groups()
            found.append({"name": name, "params": params.strip() or "(none)"})
    return found

if __name__ == "__main__":
    base, head = sys.argv[1], sys.argv[2]
    print(json.dumps(extract_signatures(changed_lines(base, head)), indent=2))

The regex is intentionally narrow because it targets public definitions only; private helpers and test code stay out of the context file. If your language is not Python, the same idea applies with a tree-sitter query or a language server instead of a regex.

Step 2: Draft with a constrained contract

The context file feeds a prompt that explicitly forbids the model from making claims beyond the provided symbols. The drafting contract below is the prompt-level enforcement of the ownership matrix.

DRAFTING_CONTRACT = """You are a documentation drafter, not an architect.
Draft only from the JSON context provided. Rules:
1. Describe only functions and classes present in the context.
2. Mark any behavior you cannot verify as TODO(verify).
3. Never invent version numbers, deprecation dates, or performance claims.
4. Output Markdown with one section per symbol.
"""

The request function is a minimal example that assumes an OpenAI-compatible chat completions shape, so verify the response format against your actual endpoint before trusting it.

import json
import os
import urllib.request

def draft(context: dict, endpoint: str, model: str) -> str:
    payload = {
        "model": model,
        "messages": [
            {"role": "system", "content": DRAFTING_CONTRACT},
            {"role": "user", "content": json.dumps(context)},
        ],
        "temperature": 0.2,
    }
    req = urllib.request.Request(
        endpoint,
        data=json.dumps(payload).encode(),
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {os.environ['API_KEY']}",
        },
    )
    with urllib.request.urlopen(req) as resp:
        return json.load(resp)["choices"][0]["message"]["content"]

This stage runs on every merge, so its cost must be near zero, and that is where free-tier access changes the economics of the pipeline. MonkeyCode's free model access fits this constraint, and the endpoint variable keeps the provider swappable if your requirements change. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 3: Schedule the drafting job on a free server

A cron job or a free CI runner invokes the script after each merge and opens a draft PR with the generated Markdown. The scheduling layer is deliberately boring because the value is in the ownership gate, not in the automation itself.

# Every hour, check for new merges and draft docs for the latest commit.
0 * * * * cd /srv/docs && ./docs_draft.py origin/main HEAD > docs/drafts/latest.md

MonkeyCode's free server option is one way to host this stage without maintaining GPU infrastructure; the script only needs an OpenAI-compatible endpoint, so moving to a different host is a one-line change. Keep the draft in a separate directory and a separate branch so the generated file never masquerades as reviewed documentation.

Step 4: Enforce the human sign-off gate

The draft PR carries a checklist that blocks merge until a human verifies each ownership category. This gate is the actual quality control; the model output is explicitly a starting point and never a mergeable artifact.

## Docs draft review
- [ ] Every symbol described exists in the current source
- [ ] No version, date, or performance claims were added by the model
- [ ] TODO(verify) items are resolved or removed
- [ ] Runnable examples were executed locally
- [ ] Architecture or security content was reviewed by a maintainer

If a reviewer cannot check the first item, the drafting contract failed and the pipeline should be fixed before the next merge. That feedback loop is what turns a free-tier drafting tool into a repeatable process rather than a one-off experiment.

Limitations and who should not use this

Free tiers carry rate limits and concurrency constraints, so batch your drafting jobs instead of streaming them per keystroke. The model can still hallucinate inside a constrained context, which means the checklist is mandatory and not a formality. Teams publishing contractual API documentation, security-sensitive projects, and repositories with no human review capacity should not adopt this workflow, because an unread draft is worse than no documentation at all.

If you want to try this pipeline this week, point the endpoint variable at a free tier and start with one module; the ownership matrix will tell you which docs are safe to automate first.

The Docs Draft Pipeline: What an AI May Write and What You Must Own

Avery Lin — Wed, 26 Aug 2026 18:51:10 +0000

The most common documentation failure is not a weak prompt or a lazy writer; it is the absence of a clear boundary between machine-draftable content and human-owned claims. A pipeline that drafts reference sections with free-tier model access and then verifies them with a symbol drift check turns docs into a testable artifact instead of a trust exercise. The model writes the inventory, and the human owns the promises.

Why documentation rots inside a healthy CI pipeline

Documentation bugs share a distinctive property: they are usually discovered by the people who consume the API, not by the pipeline that builds it. A function renamed in the last refactor stays documented under its old name until a user files an issue, and a newly added flag never appears in the docs at all. The root cause is structural, because nothing in the merge pipeline compares the documented surface against the actual code surface.

A prompt cannot know what changed inside a pull request, so the fix has to live in the pipeline around the model. The workflow drafts reference material, validates that every documented symbol still exists, and routes the remaining claims to a human reviewer. That division of labor is the entire design, and each step has a concrete tool.

The ownership boundary: what a model may draft

The first step is to separate documentation into two classes by asking a single question: can this statement be verified against the codebase alone? If the answer is yes, a model may draft it, and if the answer is no, a human must own it. The table below applies that test to the statement types that appear in most API docs.

The model may draft	A human must own
Function and class inventories	Behavioral guarantees
CLI flags and their defaults	Security and authentication properties
Config keys and their types	Compatibility and support promises
Error codes and exit statuses	Deprecation timelines
Compilable usage examples	Performance or cost claims
Parameter descriptions from signatures	Rationale for design decisions

The reason for the boundary is that normative statements cannot be checked by reading the source code alone. A deprecation promise is a commitment to future behavior, and a security property depends on the deployment environment, so neither can be derived from a function signature. When a model drafts those lines, it is guessing about commitments that only the team can make, and the reader has no way to tell the difference.

The workflow: draft, validate, classify, commit

The pipeline runs on every pull request that touches a public API, and it has four steps that map to four distinct responsibilities.

Draft the reference sections. A scheduled job reads the changed modules and produces a candidate diff for the API documentation, covering signatures, flags, and examples. For the drafting step, MonkeyCode's free model access and free server option keep this job off your laptop and out of your token budget. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Validate the symbols. The candidate diff is checked by a drift script that compares documented names against the actual AST of the package, and any mismatch fails the job before a human sees it.
Classify the claims. The diff is split by the ownership table, so the human reviewer sees only the normative lines instead of the inventory that a script already verified.
Commit with a sign-off. The merged diff includes the generated reference text plus a review line, so the documentation history records who owned each normative claim.

The drift-check artifact

The validation step is small enough to live in a single file, and it uses only the standard library. The script below extracts public symbols from a package with Python's ast module and compares them against the headings of a Markdown API document.

#!/usr/bin/env python3
"""Fail when a public symbol is undocumented or a documented symbol is stale."""
import ast
import re
import sys
from pathlib import Path


def public_symbols(root: Path) -> dict:
    symbols = {}
    for path in root.rglob("*.py"):
        if any(part.startswith(".") for part in path.parts):
            continue
        tree = ast.parse(path.read_text(), filename=str(path))
        for node in tree.body:
            if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
                if node.name.startswith("_"):
                    continue
                symbols[f"{path.stem}.{node.name}"] = node.lineno
    return symbols


def documented_symbols(doc: Path) -> set:
    text = doc.read_text()
    return set(re.findall(r"^#{2,3}\s+`?([\w.]+)`?", text, re.M))


def main() -> int:
    root = Path(sys.argv[1] if len(sys.argv) > 1 else "src")
    doc = Path(sys.argv[2] if len(sys.argv) > 2 else "docs/api.md")
    actual = public_symbols(root)
    documented = documented_symbols(doc)
    missing = sorted(set(actual) - documented)
    stale = sorted(documented - set(actual))
    for name in missing:
        print(f"UNDOCUMENTED {name}")
    for name in stale:
        print(f"STALE {name}")
    return 1 if missing or stale else 0


if __name__ == "__main__":
    sys.exit(main())

Run it locally or inside a CI step with a single command, and the exit code drives the merge decision for the documentation diff.

python doc_drift.py src docs/api.md

- name: Check documentation drift
  run: python doc_drift.py src docs/api.md

The script is deliberately naive, because a simple check that runs on every PR is worth more than a sophisticated one that nobody maintains. It expects one documented symbol per heading, so your API document needs a consistent heading format, and it only inspects top-level module symbols, so nested classes require a small extension.

What the script cannot see

The drift check proves that a documented name exists, but it cannot prove that the description next to that name is true. A model can confidently draft a parameter explanation that matches the signature while being wrong about the actual behavior, and no symbol comparison will catch it. That gap is exactly why the ownership table routes behavioral claims to a human reviewer instead of trusting the draft.

Free-tier generation also has practical limits that shape the design, and you should plan around them from the start. Treat the drafting job as a scheduled batch process rather than an interactive assistant, because a per-PR batch tolerates rate limits and retries far better than a blocking review step. The free server option matters for the same reason, since an unattended job in CI should not depend on a developer's local machine being online.

Who should not use this workflow

Teams without a designated reviewer should not adopt this pipeline, because the human ownership step is the part that prevents false documentation, not the model. Projects whose docs are mostly narrative, such as design histories or migration guides, will get little value from a symbol inventory, and the drift check will only add noise. If your documentation is already generated from a single source like OpenAPI or a type system, the script duplicates work that your generator already performs.

The ownership boundary is the transferable idea here, and the script is just a minimal implementation of one half of it. Run the drift check on your repository for a month and count the false positives, because that number tells you how much trust your documentation actually deserves.

The Draft/Own Split: A Reproducible Workflow for Generated Docs

Avery Lin — Wed, 26 Aug 2026 15:46:39 +0000

The Draft/Own Split: A Reproducible Workflow for Generated Docs

Documentation generation fails less often because the model writes poorly and more often because nobody defined who owns the claims inside the generated text. A model can draft API references, quickstarts, and migration notes, but architecture rationale, security guarantees, and compatibility promises must stay with a human owner. This article defines a draft/own split, provides a small audit script, and shows where a free tier fits without becoming the owner of record.

The failure mode is ownership, not grammar

Most teams evaluate generated documentation by reading it, and that habit measures fluency instead of correctness. A doc comment that reads smoothly can still encode a wrong precondition, an outdated default, or a promise the code never made. The real question is whether someone with authority verified each claim, and that question never appears in the rendered output.

A practical signal exposes the problem quickly: track how many doc comments change in the same pull request that changes the code. If the docs never move when the code moves, the generation step produces decoration rather than documentation. If the docs move but no human reviewed the diff, the generation step produces unverified claims that later become integration failures.

The workflow

The draft/own split runs in five numbered steps, and each step names a single owner for its output.

Step 1: Inventory the API surface. Run a script that lists every exported symbol, its signature, and whether it carries a doc comment. This converts "we should document more" into a countable gap that the team can track over time.

Step 2: Classify each doc type. Apply the decision matrix below to decide whether the model drafts, the human owns, or both participate in the final text.

Step 3: Generate drafts for the low-risk classes. Let the model produce the first version of function references, quickstarts, and changelog entries, then review the diff before it reaches the main branch. I ran this step with MonkeyCode's free model access on their free server, so the drafting cost nothing for a small repository. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 4: Human owns the high-risk classes. Architecture decisions, security notes, and compatibility promises get written or rewritten by a human, and the model may only copy-edit afterward.

Step 5: Verify with the audit script. Re-run the inventory, confirm that every exported symbol has a doc comment, and route any claim words to a human check.

The decision matrix

Doc type	Risk if wrong	Audience	Owner
Function and class references	Medium	Other developers	Model drafts, human verifies
Quickstart and setup guides	Low	New users	Model drafts, human verifies
Migration notes	High	Existing users	Human owns
Architecture rationale	High	Maintainers	Human owns
Security and permissions	Critical	Operators	Human owns
Changelog entries	Medium	All users	Model drafts from git log, human verifies

The rule is simple: the model may draft anything a human can verify in under five minutes. The human owns anything whose failure sends a reader down the wrong integration path. Risk drives the split rather than the model's confidence, because confidence is not evidence.

The audit script

The script below inventories exported symbols in a CommonJS project and flags every export that lacks a JSDoc comment. It is deliberately small so you can read it in one pass and adapt it to your own conventions. A TypeScript variant would use the compiler API to resolve exported declarations, but the CommonJS version keeps the logic readable.

// doc-audit.mjs
import { readdirSync, readFileSync } from "node:fs";
import { join, extname } from "node:path";

const root = process.argv[2] ?? "src";
const files = [];

function walk(dir) {
  for (const entry of readdirSync(dir, { withFileTypes: true })) {
    const full = join(dir, entry.name);
    if (entry.isDirectory()) walk(full);
    else if (extname(entry.name) === ".js") files.push(full);
  }
}

walk(root);

let missing = 0;
for (const file of files) {
  const source = readFileSync(file, "utf8");
  const lines = source.split("\n");
  for (let i = 0; i < lines.length; i++) {
    const match = lines[i].match(/export\s+(?:function|const|class)\s+(\w+)/);
    if (!match) continue;
    const before = lines.slice(Math.max(0, i - 10), i).join("\n");
    const hasDoc = /\/\*\*[\s\S]*?\*\//.test(before);
    if (!hasDoc) {
      missing++;
      console.log(`${file}:${i + 1} missing doc for ${match[1]}`);
    }
  }
}

console.log(`Total missing: ${missing}`);

Run it with node doc-audit.mjs src and the output lists every undocumented export with its file and line number. After the model drafts the missing comments, run the script again and the count should drop to zero for the draft-classified items.

The second verification pass targets claim words instead of mere presence, because a doc comment can exist and still be wrong. Grep the generated docs for terms like always, never, guaranteed, secure, and thread-safe, then send every match to the human owner. These words carry promises that a fluent sentence can hide, and they are exactly the claims the decision matrix reserves for human review. Run the audit script on your own repository this week, and you will see exactly where your documentation debt lives.

Where the free tier fits

The free tier from Step 3 made the drafting step affordable for a side project, but the economics are not the point of this workflow. The point is that the model produces a first version, the human owns the claims, and the audit script closes the loop. If you have no budget at all, the same workflow works with any local model, and the ownership rules remain unchanged.

Limitations

This workflow assumes a codebase with an explicit public API, so it fits libraries and services with clear boundaries. It does not fit monorepos with dozens of packages, because the inventory step needs a package-aware traversal that this script does not implement. It also does not fit primarily conceptual documentation such as tutorials, since a narrative arc cannot be captured by an ownership matrix. The audit script checks presence, not correctness, and presence is a necessary condition rather than a sufficient one.

Who should not use this

Teams without a human reviewer should not treat this split as a substitute for review, because the workflow degrades into unverified generation with a checkbox. Small scripts with a handful of functions may not need the inventory step, since the overhead outweighs the benefit when the API surface fits on one screen. If your team already skips code review under deadline pressure, it will skip this human step too, and the matrix becomes theater.

Draft with a Model, Sign with a Human: A Documentation Ownership Workflow

Avery Lin — Wed, 26 Aug 2026 12:43:47 +0000

Documentation quality is decided by the ownership boundary before the prompt is written, not by the model's fluency. A model can draft descriptions, signatures, and happy-path examples, but it cannot know which invariants your team is willing to defend. This article defines a workflow where the model produces a first draft and a human explicitly signs the claims that matter.

The failure mode

Most documentation pipelines treat generation as a text problem: feed in code, get prose, publish. The result is a page that reads well and contains no obvious lies, yet still fails because it omits the one constraint that changes behavior. A common failure appears in API references where the generated text describes parameters but never states that a null value disables retries. That sentence is not a style improvement; it is a correctness requirement.

The ownership matrix

Before generating anything, write down what the model may own and what a human must own. The table below is a starting point that works for libraries, CLIs, and internal services.

Artifact	Model may draft	Human must own
Function signatures and parameter names	Extracted from the AST	The semantic meaning of null, empty, and default values
Usage examples	Derived from existing tests	Which examples are supported and which are accidental
Invariants and failure modes	Proposed candidates	Confirmation that each invariant is true and testable
Deprecation and deletion criteria	A suggested timeline	The actual removal policy and migration plan
Decisions and rationale	A summary of code comments	The reasoning behind non-obvious choices

The workflow

Use this workflow when you need to ship documentation for a module that already has tests and a stable public surface. It takes about an hour for a typical package and produces a file that CI can check.

1. Define the contract

Create a markdown template with explicit ownership markers. Every section gets either  or ; the second type requires a Reviewed-by line before merge. This marker is the contract, and the prompt is just the instruction that fills it.

# docs/template.md
<!-- OWNER: model -->
## Overview

<!-- OWNER: model -->
## API reference

<!-- OWNER: human -->
## Invariants
<!-- Reviewed-by: -->

<!-- OWNER: human -->
## Deprecation policy
<!-- Reviewed-by: -->

2. Generate the draft

Feed the template and the module's public symbols into a generation pass. In this workflow, MonkeyCode's free model access and free server option provide a low-cost way to run that pass in a disposable environment; the same steps work with any provider. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Keep the prompt narrow: ask for descriptions, not decisions.

# generate_docs.py
from pathlib import Path
import subprocess

def public_symbols(path: Path) -> str:
    # Simplified: grep for top-level definitions.
    return subprocess.run(
        ['grep', '-nE', '^(def|class) ', str(path)],
        capture_output=True, text=True
    ).stdout

def build_prompt(template: Path, source: Path) -> str:
    return f'''
{template.read_text()}

Generate documentation for the public symbols below.
Do not invent invariants. Mark every unverified claim as UNVERIFIED.

{public_symbols(source)}
'''

3. Lint the ownership

Run a small script that fails when a human-owned section has no reviewer. This lint does not check truth; it checks that someone has claimed responsibility for the truth.

# doc_ownership.py
import re
import sys
from pathlib import Path

HUMAN_OWNED = re.compile(r'<!--\s*OWNER:\s*human\s*-->', re.I)
REVIEWED = re.compile(r'<!--\s*Reviewed-by:\s*(.+?)\s*-->', re.I)

def check(path: Path) -> list[str]:
    text = path.read_text()
    errors = []
    for i, line in enumerate(text.splitlines(), 1):
        if HUMAN_OWNED.search(line) and not REVIEWED.search(line):
            errors.append(f'{path}:{i}: human-owned section lacks Reviewed-by')
    return errors

if __name__ == '__main__':
    problems = [e for p in sys.argv[1:] for e in check(Path(p))]
    if problems:
        print('\n'.join(problems))
        raise SystemExit(1)
    print('ownership contract satisfied')

4. Review the human-owned sections

Read only the sections marked human-owned, not the whole page. For every invariant, ask two questions: is this true today, and will CI catch it if it changes? If the second answer is no, delete the sentence or add a test that protects it.

5. Measure the handoff

Track two numbers: time from draft to merge and the number of corrections made to human-owned sections. If corrections stay high, the contract is wrong, not the model. Adjust the matrix until the model drafts only what your reviewers can verify quickly.

Limitations and who should not use this

A model cannot know the hidden constraints that live in issue comments, incident reviews, or the memory of the person who wrote the code. This workflow also assumes that a human reviewer exists; a team without a designated doc owner will skip the signing step and recreate the original problem. If your documentation must satisfy regulatory traceability, use a formal review system instead of markdown comments. The free model access and free server option are useful for experiments, but they are not a substitute for the ownership boundary.

Try this on one module's README and compare the review time against your previous process. The boundary, not the model, will tell you where the real cost lives.

Triage CI Failures with a Free-Tier Model: A Reproducible Classifier Pipeline

Avery Lin — Wed, 26 Aug 2026 11:41:58 +0000

A red build stops your team. Someone must read the logs. Someone must decide: flaky, regression, or environment? That decision is mechanical most of the time. A small model can do it. This article builds a failure classifier. It runs on free models and a free server. You get a script you can run today.

The hidden cost of failure triage

Every red CI run interrupts flow. The first responder reads logs, checks the diff, and guesses. Most failures fall into four buckets. The guess is usually right, but it still costs ten minutes. Multiply that by every developer and every broken build.

Automation rarely helps because the tooling is too heavy. Full observability platforms need setup. Custom classifiers need labeled data. A free-tier model needs neither.

Why a small model is enough

Failure triage is a classification task with short inputs. The log is a few hundred lines. The diff is a few files. The output is one label and a short justification. This is not a reasoning-heavy job. A free model handles it well.

MonkeyCode is an open-source project that fits this exact spot. It offers free models and a free server option. The pipeline below runs at zero inference cost. The script uses a generic CLI adapter, so you can swap the backend later.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The decision table

Before writing code, define the labels. Use five buckets. Keep them mutually exclusive.

Label	Signal	Action
flaky	Test failed once, no related diff	Re-run in CI
regression	Diff touches the failing path	Assign to author
environment	Dependency, network, or timeout	Re-run, then escalate
test_bug	Assertion or fixture error	Fix the test
unknown	No clear signal	Manual review

The table is the prompt. The model maps evidence to a label. The evidence field is what you verify.

The classifier script

The script below reads a CI log and a diff. It sends both to the model. It expects a JSON response with three fields: label, confidence, and evidence.

#!/usr/bin/env python3
"""triage.py — classify a CI failure with a free-tier model."""
import json
import subprocess
import sys
from pathlib import Path

LOG_PATH = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("ci.log")
DIFF_PATH = Path(sys.argv[2]) if len(sys.argv) > 2 else Path("diff.txt")

LABELS = {
    "flaky": "Test failed once, no related diff.",
    "regression": "Diff touches the failing path.",
    "environment": "Dependency, network, or timeout.",
    "test_bug": "Assertion or fixture error.",
    "unknown": "No clear signal.",
}

PROMPT = f"""
You are a CI failure triage assistant.
Read the log and the diff. Choose exactly one label.
Labels: {json.dumps(LABELS)}
Return JSON: {{"label": "...", "confidence": 0.0-1.0, "evidence": "..."}}
Evidence must quote the log. Do not invent facts.
"""

def run_cli(prompt: str) -> str:
    """Send a prompt to the model CLI via stdin and return stdout."""
    result = subprocess.run(
        ["mc", "--free-server", "prompt"],
        input=prompt,
        capture_output=True,
        text=True,
        check=True,
    )
    return result.stdout

def main() -> None:
    log = LOG_PATH.read_text()[-4000:]
    diff = DIFF_PATH.read_text()[-2000:]
    response = run_cli(PROMPT + f"\n\nLOG:\n{log}\n\nDIFF:\n{diff}")
    try:
        data = json.loads(response)
    except json.JSONDecodeError:
        print("{\"label\": \"unknown\", \"confidence\": 0.0, \"evidence\": \"Model output was not JSON.\"}")
        return
    print(json.dumps(data, indent=2))

if __name__ == "__main__":
    main()

The script truncates inputs. Long logs lose context, but the tail usually contains the failure. The diff tail contains the most recent changes.

Prompt design notes

The prompt has three deliberate constraints. First, it forces one label. A model that can answer "unknown" will overuse it. Second, it demands quoted evidence. That makes the output verifiable. Third, it forbids invented facts. That keeps the model honest.

You can tune the labels later. Some teams split regression into logic and data. Some add a migration label. The structure stays the same: label, confidence, evidence.

Run it locally first

Before touching CI, run the script on a saved failure. The command is simple:

python triage.py ci.log diff.txt

You should see JSON on stdout. If the model returns prose instead, add a system prompt that demands JSON. If the output is still malformed, fall back to the unknown label and review manually.

Example output

A healthy run returns clean JSON. Here is a realistic response:

{
  "label": "regression",
  "confidence": 0.87,
  "evidence": "test_auth.py:41 failed; diff changes validate_token() signature."
}

The evidence is the part you trust. The label is a hypothesis. The confidence is a hint, not a guarantee.

A second run might return a different verdict:

{
  "label": "flaky",
  "confidence": 0.74,
  "evidence": "test_orders.py:22 failed; diff touches billing.py only."
}

That is the case where the model earns its place. It reads the diff, finds no connection, and says so. A tired developer might have missed that.

Run it in CI

Add the script to your pipeline. Use three numbered steps.

Capture the failing log and the diff before the job exits.
Run the classifier and write the JSON to a file.
Post the label and evidence as a comment on the commit.

A minimal CI snippet looks like this:

- name: Capture failure
  if: failure()
  run: |
    tee ci.log
    git diff origin/main...HEAD > diff.txt
- name: Classify failure
  if: failure()
  run: python triage.py ci.log diff.txt > triage.json
- name: Comment result
  if: failure()
  run: gh pr comment "$PR" --body "$(cat triage.json)"

The comment gives the next developer a starting point. It does not replace judgment. It removes the first ten minutes of reading.

Validate the buckets first

Do not wire this into CI immediately. Run it on last week's failures first. Compare the model labels with what your team actually decided. Track three numbers: precision on regression, precision on flaky, and the unknown rate.

Metric	Target
Regression precision	> 0.8
Flaky precision	> 0.7
Unknown rate	< 0.3

If the unknown rate is above 30 percent, your buckets do not match your logs. Adjust the labels and try again.

Validation is cheap. It costs one script run and a few minutes of reading. It prevents the worst outcome: a confident model that is confidently wrong.

When the classifier lies

The model will be wrong. Treat the label as a suggestion. The evidence field is the real deliverable. It points at the log lines that matter.

Do not use this for security incidents. Do not use it for on-call pages. Do not let it auto-close issues. The pipeline is a triage aid, not an authority.

Who should not use this

Teams without a human review step will trust the label blindly. That is worse than no automation. Regulated environments need a documented decision trail, not a model summary. If your CI logs contain secrets, do not send them to any remote model. The free server option changes cost, not data handling.

Try it with zero cost

The full pipeline costs nothing to run. MonkeyCode's free models and free server cover the inference. The script is a template. Adapt it to your CI system and your failure patterns.

Start with last week's failures. Run the classifier on old logs. Compare its labels with what your team decided. That validation tells you whether the buckets fit your reality. If they do, wire it into CI and reclaim the first ten minutes of every red build.

Docs Generation Needs a Handoff Contract, Not a Better Prompt

Avery Lin — Wed, 26 Aug 2026 09:03:01 +0000

AI documentation tools can draft accurate reference prose from code, but the boundary between machine-drafted and human-owned content is a contract, not a style preference. A prompt that produces clean prose on Monday will produce confident nonsense on Tuesday, because the model has no persistent memory of your invariants or your audience. The fix is not a longer system prompt; it is a workflow that separates mechanically verifiable claims from claims that require human judgment.

Generated code fails loudly at runtime, while generated documentation fails quietly in a design review six months later. A wrong parameter table breaks a build immediately, but a wrong rationale becomes the basis for a future decision that nobody can trace. This asymmetry is why documentation generation deserves the same discipline as code review: a reproducible pipeline, a small review surface, and an explicit owner for every claim.

The Handoff Contract

The first step is to define what the model may draft and what a human must own. For content that can be checked against the source tree, the model is reliable: API signatures, parameter tables, type annotations, default values, and changelog bullets from commit messages. The model is not reliable for content that states intent: design rationale, rejected alternatives, security decisions, deprecation promises, and compatibility guarantees. Those claims have no mechanical ground truth, so they require a named human owner.

Artifact	Model draft	Human owns	Verification
API parameter table	Yes	Type and default accuracy	AST extraction against source
Changelog entry	Yes	Severity and target audience	Commit message review
Migration note	Draft	Compatibility promise	Manual sign-off
Design rationale	No	Yes	Review meeting
Security note	No	Yes	Threat model review

The table above is the contract that the pipeline enforces. Every generated document should carry a manifest that classifies each claim as verifiable or judgment. The pipeline should refuse to merge until each judgment claim has an approver.

A Reproducible Handoff Pipeline

The artifact below is a small extractor that turns a Python module into a documentation draft plus a claim manifest. It does not generate prose; it generates the structure that a model can fill and the checklist that a human must sign. The script uses only the standard library, so it runs in any CI environment without extra dependencies.

#!/usr/bin/env python3
"""Emit a doc draft and a claim manifest for human sign-off."""
import ast
import json
import sys
from pathlib import Path

def extract_public_api(path: Path):
    tree = ast.parse(path.read_text())
    nodes = []
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            if not node.name.startswith("_"):
                nodes.append(node)
    return nodes

def emit_draft(nodes):
    lines = ["# API Reference Draft", ""]
    for node in nodes:
        lines.append(f"## {node.name}")
        lines.append("")
        lines.append("_Draft: verify behavior before merge._")
        lines.append("")
        if isinstance(node, ast.FunctionDef):
            args = [a.arg for a in node.args.args]
            lines.append(f"Parameters: {', '.join(args) or 'none'}")
        lines.append("")
    return "\n".join(lines)

def emit_claims(nodes):
    claims = []
    for node in nodes:
        claims.append({
            "claim": f"{node.name} is public and extracted from source",
            "kind": "verifiable",
            "check": "AST extraction",
        })
        claims.append({
            "claim": f"{node.name} behavior matches the generated text",
            "kind": "judgment",
            "owner": "unassigned",
        })
    return claims

if __name__ == "__main__":
    src = Path(sys.argv[1])
    api = extract_public_api(src)
    Path("docs").mkdir(exist_ok=True)
    Path("docs/draft.md").write_text(emit_draft(api))
    Path("docs/claims.json").write_text(
        json.dumps(emit_claims(api), indent=2)
    )

Run it on any module and you get two files: a draft with placeholders and a JSON manifest with exactly two claims per symbol. The model fills the draft while the human approves the manifest, and the pipeline then enforces the contract in CI. The manifest is the review surface, so the reviewer reads a dozen claims instead of five hundred lines of prose.

The Workflow in Five Steps

The workflow runs in five steps, and each step has a single owner. The first three steps are automated; the last two require a human decision. Keeping the steps separate prevents the model from silently owning a claim it cannot verify.

Extract. Run the script on every changed module in a pull request and commit the resulting skeleton to the docs directory.
Generate. Send the skeleton to a model with a constrained instruction: fill the parameter tables from the signatures, and do not add rationale or promises.
Merge. The model's output replaces the placeholders in docs/draft.md, while the manifest stays untouched.
Verify. A CI check fails the build when any judgment claim still has "owner": "unassigned".
Approve. A human reviews only the judgment claims, which is a much smaller surface than the full prose.

The verification step is a short script that any CI runner can execute:

python docgen_handoff.py src/payments.py
python - <<'PY'
import json, sys
claims = json.load(open("docs/claims.json"))
unowned = [c for c in claims if c["kind"] == "judgment" and c["owner"] == "unassigned"]
if unowned:
    print("Unapproved judgment claims:", [c["claim"] for c in unowned])
    sys.exit(1)
PY

The generation step needs a model endpoint, and metered APIs make that step expensive when you run it on every pull request. MonkeyCode's free model access and free server option cover the generation step without a per-token bill, which makes per-PR doc regeneration practical in the first place. The workflow itself is model-agnostic, so the contract survives any change in the underlying provider. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the Pipeline Cannot Verify

The extractor knows structure, not semantics, so it cannot confirm that a docstring matches runtime behavior. The manifest catches missing approvals, but it cannot catch a wrong claim that a human approves in a hurry. Free access tiers also change over time, so check the current terms before you build a pipeline that depends on them.

Who Should Not Use This Workflow

Teams without a human reviewer for judgment claims should not adopt this pipeline, because the manifest becomes a rubber stamp. Projects where documentation is the product, such as SDK guides or compliance manuals, need a voice and a provenance model that a generated draft cannot provide. Regulated environments that require audit trails for every published statement should keep generation out of the critical path entirely.

The Measure That Matters

Run the extractor on one module and count how many judgment claims survive review with edits. If the number is high, your documentation debt lives in the rationale, not the prose, and no prompt will fix it. If the number is low, the model is doing the mechanical work and your reviewers are doing the thinking. That is the division of labor this contract was designed to enforce.

Opinion: Your AI Reviewer Should Be a Background Process, Not a Merge Gate

Avery Lin — Wed, 26 Aug 2026 05:50:09 +0000

The blocking AI review gate is the wrong abstraction for most teams, and the evidence is in how teams actually use it. A gate runs once, late, under merge pressure, and every question it asks costs either money or latency. The result is a predictable rationing behavior: developers ask the reviewer only the questions they can afford, which are rarely the questions that matter. The better model is a background loop that watches every push and runs cheap checks continuously, with escalation reserved for actual risk signals.

A lot of the current discussion about AI-assisted review focuses on the reviewer's competence, and almost none of it focuses on the reviewer's schedule. Nobody tested the reviewer, but more importantly, nobody designed the schedule on which the reviewer works. A reviewer that appears once per pull request behaves like an auditor, while a reviewer that runs on every commit behaves like a monitoring agent. Those two roles produce different findings, and the monitoring role is the one that catches issues before they become merge debates.

The gate model forces question rationing

Think about what a blocking gate actually does to your review budget. Each pull request gets one pass, usually triggered by a human who is already late, and that pass has a fixed cost in tokens, latency, or both. When the cost is metered, the rational move is to trim the prompt, skip the follow-up question, and accept the first answer. That trimming is exactly where review quality dies, because the second and third questions are the ones that probe assumptions.

The free-tier argument still holds, but the architecture matters more than the price. Free inference removes the per-question tax, yet a gate still limits the number of questions by construction. You get one shot per pull request, and the model never sees the conversation that happens after the merge. A background loop removes that structural limit because it can ask questions continuously, on every push, without a human in the loop.

The loop model is a triage pipeline, not a chatbot

The position here is simple: run review like an observability pipeline, with a cheap tier that always runs and an expensive tier that escalates on signal. This is not a new idea in operations, where metrics feed alerts and alerts feed pages, and it transfers cleanly to code review. The cheap tier classifies the diff, scores its risk, and flags structural patterns like deleted lines in core paths. The expensive tier runs only when the cheap tier produces a signal, which keeps the average cost near zero.

This is also not the per-commit review argument restated, because a loop is not a gate that runs more often. A per-commit gate still blocks, still waits for a verdict, and still interrupts the author, while a loop writes findings asynchronously and lets the author continue. The schedule change is the point: review becomes a background process instead of a ceremony.

MonkeyCode's free server option and free model access fit this pattern because they remove the two costs that usually kill it. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A free server gives the loop a permanent home that is not your laptop, and free models make the always-on cheap tier financially boring. Boring is the goal, because a review loop that requires budget approval every month will be switched off in the first quarter.

Building the loop in five steps

The workflow below is a minimal version worth running on a side project before it touches a production repo. Each step produces a concrete artifact, and the order matters because the escalation rule depends on the cheap tier being reliable. If a step feels optional, that is the step your loop will silently skip later.

Provision a free server and a free model as the default tier. The server runs the loop, and the model answers the cheap questions; both need to be boringly available.
Define the cheap-check battery as a fixed list of questions. A sensible default battery is a diff summary, a risk score from zero to one, a deleted-line audit, and a test-gap hint.
Write the loop as a small daemon that polls the repository and reacts to new commits. The script below is a skeleton, not production code, and the model call is a placeholder for whatever endpoint your provider documents.
Add an escalation rule that routes high-risk diffs to a second pass. The threshold should be calibrated on your own history, not on a vendor's benchmark, because risk is repo-specific.
Post findings asynchronously and never block the merge on the cheap tier. The loop informs, and the human decides, which preserves the review conversation instead of replacing it.

The decision table that keeps the loop honest

Check	Default tier	Escalation trigger	Output
Diff summary	free model	none	PR comment
Risk score	free model	score >= 0.7	expensive pass
Deleted-line audit	free model	core path, > 10 deletions	human ping
Test-gap hint	free model	none	draft test list

The escalation column is the part most teams skip, and it is the part that prevents the loop from becoming noise. If every diff gets the expensive pass, you have rebuilt the gate with extra steps. If no diff gets it, the loop is decorative.

The skeleton loop

#!/usr/bin/env python3
"""Background review loop: cheap checks first, escalate on signal."""
import subprocess
import time
from pathlib import Path

REPO = Path("/srv/app")
RISK_THRESHOLD = 0.7

def last_commit():
    return subprocess.check_output(
        ["git", "-C", str(REPO), "rev-parse", "HEAD"], text=True
    ).strip()

def diff_for(commit):
    return subprocess.check_output(
        ["git", "-C", str(REPO), "show", commit, "--stat"], text=True
    )

def cheap_review(diff_text):
    # Placeholder: call the free-model endpoint here.
    # Returns (summary, risk_score) where risk_score is 0.0-1.0.
    return {"summary": diff_text[:200], "risk_score": 0.4}

def expensive_review(diff_text):
    # Placeholder: escalation pass, run only when risk is high.
    return {"verdict": "needs human eyes", "notes": []}

def main():
    seen = last_commit()
    while True:
        time.sleep(60)
        current = last_commit()
        if current == seen:
            continue
        diff = diff_for(current)
        result = cheap_review(diff)
        print(f"[review] {current}: {result['summary']}")
        if result["risk_score"] >= RISK_THRESHOLD:
            print(f"[review] escalation: {expensive_review(diff)}")
        seen = current

if __name__ == "__main__":
    main()

The polling interval and the risk function are the two knobs you will tune first. Start with sixty seconds and a conservative threshold, then measure how many escalations actually changed a merge decision. The loop is deliberately small because the failure mode of this pattern is complexity, not coverage.

Who should not use this approach

Teams with a compliance requirement for documented, blocking review should keep the gate. Regulated environments need an audit trail that says a named reviewer approved a specific revision, and a background loop cannot produce that. Small repos with very few pull requests also gain little, because the setup cost exceeds the review cost you are trying to save.

The free tier also has limits, and pretending otherwise is how loops die. Long context windows, deep architectural reasoning, and multi-file refactor audits are exactly the cases where the expensive tier should take over. The loop is a triage system, not a replacement for judgment, and the escalation rule is the part that acknowledges that.

Try the loop on a side project first, and measure how many questions you skip per week under the gate model. The cost of being wrong in that setting is a comment, not a merge, which is the right place to learn the difference. Once the loop earns trust, the gate can be retired, and review becomes something the repository does rather than something the team schedules.

Opinion: An AI Review You Can't Replay Is a Review You Can't Trust

Avery Lin — Wed, 26 Aug 2026 02:48:16 +0000

Most teams treat AI review output like a chat message: read it, act on it, and forget it. The correct mental model is test output, which means it must be re-runnable, comparable, and archived. A review you cannot replay is a review you cannot audit, and an unauditable review is worse than none because it manufactures confidence. My position is that replayability is the missing quality gate in the AI review loop, and that free model access finally makes it affordable.

The three failures that replay exposes

Review pipelines fail in three recurring ways, and all three stay invisible until you try to re-run the review.

The unexplainable false positive. A model flags a refactor as risky, a human spends an hour investigating, and the finding turns out to be noise. When the review is ephemeral, you cannot reconstruct what the model saw, so you cannot learn whether the prompt or the diff caused the noise.
Silent config drift. Someone edits the system prompt or bumps the model, and the review behavior shifts underneath the team. Nothing fails, no test breaks, and the merge discipline changes without a single deliberate decision.
The provider swap. A hosted model is replaced upstream, verdicts flip on identical diffs, and your gate suddenly blocks more or less than it did last week. Without a replay, the change looks like a change in code quality when it is actually a change in the reviewer.

The recent DEV conversation about AI reviewers never being tested points at the same gap from a different direction. We test the code the reviewer produces, but we never test the reviewer itself, and replay is the cheapest test that exists.

Why tests get re-run and reviews do not

A unit test is valuable because you can run it again, and a failing re-run tells you the code changed. AI review output is stochastic, so the contract is weaker: identical inputs can produce different outputs, which means you must snapshot more, not less. The discipline is to archive every input that shaped the review — the diff, the prompt, the model identifier, and a hash of the review configuration — alongside the output. That archive turns a one-off opinion into a reproducible record, and it turns drift into a detectable event.

A replay harness you can run today

The artifact below is a minimal replay harness that assumes your review server exposes an HTTP endpoint and that you archive each review as a JSON record. It re-runs an archived review, compares the verdict, and then diffs the full output. Adapt the payload to your own API, since the exact field names will vary.

#!/usr/bin/env bash
set -euo pipefail

RECORD="${1:-reviews/latest.json}"
BASE_URL="${REVIEW_SERVER_URL:-http://127.0.0.1:8787}"

DIFF_FILE="$(jq -r .diff_file "$RECORD")"
PROMPT_FILE="$(jq -r .prompt_file "$RECORD")"
MODEL="$(jq -r .model "$RECORD")"
CONFIG_HASH="$(jq -r .config_hash "$RECORD")"
ARCHIVED_OUTPUT="$(jq -r .output_file "$RECORD")"

curl -sS -X POST "$BASE_URL/review" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --rawfile diff "$DIFF_FILE" --rawfile prompt "$PROMPT_FILE" \
        --arg model "$MODEL" --arg config_hash "$CONFIG_HASH" \
        '{diff: $diff, prompt: $prompt, model: $model, config_hash: $config_hash}')" \
  > /tmp/replay_output.json

NEW_VERDICT="$(jq -r .verdict /tmp/replay_output.json)"
OLD_VERDICT="$(jq -r .verdict "$ARCHIVED_OUTPUT")"

if [[ "$NEW_VERDICT" != "$OLD_VERDICT" ]]; then
  echo "REPLAY MISMATCH: verdict changed from $OLD_VERDICT to $NEW_VERDICT"
  exit 1
fi

if ! diff <(jq -S . "$ARCHIVED_OUTPUT") <(jq -S . /tmp/replay_output.json) > /tmp/replay.diff; then
  echo "REPLAY DRIFT: output changed for identical inputs"
  cat /tmp/replay.diff
  exit 1
fi

echo "REPLAY OK: output matches the archived review"

The workflow has four steps, and each one maps to a normal engineering habit.

Archive the record. After every review, write a JSON record that references the diff file, the prompt file, the model identifier, and a hash of the review configuration. The record is the unit of audit.
Replay before merge. Run the harness on the review for the current pull request, and treat a verdict change as a blocker. A reviewer that contradicts itself on identical input needs a human decision, not a silent coin flip.
Replay on config change. When you edit the prompt or switch models, replay the last ten archived reviews and measure how many verdicts flip. That number is the blast radius of your change, and it belongs in the pull request description.
Replay on a schedule. A weekly cron job that replays a sample of archived reviews turns drift into a trend. The job does not need to block anything; it exists to make invisible changes visible.

Why free execution changes the incentive

Replay only becomes a habit when the marginal cost of a re-run is near zero. Metered review APIs punish exactly the behavior this workflow requires, because running the same input twice means paying for the same work twice. Free model access and a free server option invert that incentive, since the marginal cost of a replay stops being the dominant term in the decision. MonkeyCode offers both, which makes the replay harness a default rather than a luxury. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The economics matter because discipline follows cost. When a re-run is free, the archive stops being a compliance artifact and starts being a debugging tool, and teams actually use debugging tools.

Limitations and who should skip this

This workflow is not a determinism guarantee, and I want to be explicit about that. Temperature and sampling mean identical inputs can still produce different outputs, so the harness detects drift rather than eliminating it, and a passing replay is evidence of stability, not proof of correctness. The approach also assumes your review server can be pointed at archived inputs, which is a property of the server implementation rather than the model. Teams with a handful of weekly reviews will find the archive overhead larger than the benefit, and teams whose compliance rules require a hosted vendor in the loop cannot use a local replay at all.

The honest conclusion is that replayability is a trust mechanism, not a quality mechanism. It will not make your AI reviewer smarter, but it will tell you when the reviewer changes, and that knowledge is the precondition for every other improvement you might attempt. If your pipeline already archives diffs, replay the last ten reviews this week; the mismatches will teach you more about your reviewer than the reviews ever did.

Opinion: Your AI Reviewer Needs a Regression Suite, Not a Leaderboard Score

Avery Lin — Tue, 25 Aug 2026 23:28:54 +0000

Choosing an AI code reviewer by public benchmark is the same mistake as choosing a linter by its README. Your repository has failure modes that no general benchmark can predict, and those failures are exactly what your reviewer must catch. The fix is small and versioned: a regression suite of ten past pull requests, run every time your review prompt, model, or configuration changes.

Why Leaderboard Scores Mislead You

Public benchmarks rank models on curated coding tasks, but your review pipeline runs against your diff history. A model that scores well on generic coding tasks can still miss the null-check pattern your team forgets every quarter. Review quality also depends on the prompt, the diff context, and your repository conventions, so a benchmark number tells you little about your pipeline.

The deeper problem is that review is a comparative task, not a generative one. A reviewer is useful only when it catches defects your team would otherwise merge, and tolerable only when it does not flood PRs with noise. Both properties are local to your codebase, which means the only honest evaluation is one built from your own history.

The Fix: A Ten-PR Regression Suite

Treat your AI reviewer like any other dependency you maintain. Pinning its configuration is necessary but not sufficient, because a pinned reviewer can still drift in behavior when the underlying model changes. Collect a small set of historical pull requests with known outcomes, define a pass criterion for each, and run the suite on every configuration change. Ten PRs is a heuristic, not a statistical guarantee, but it catches the most damaging regressions without becoming a maintenance burden.

Step 1: Collect Ten Historical Pull Requests

Pick five PRs that contained a real defect your team caught in review, and five clean PRs that merged without issues. The defective PRs should represent your most common bug classes, not the most dramatic ones. Store each diff as a file, and record the defect location plus the expected review comment in a small JSON manifest.

Step 2: Define Pass Criteria

For each defective PR, the reviewer must flag the defect location with a comment a human would recognize as actionable. For each clean PR, the reviewer must not block the merge with a false positive. Write these expectations in the manifest so the suite can be evaluated by a script or a human in under ten minutes.

{
  "regressions": [
    {
      "id": "pr-1234",
      "diff": "regression/diffs/pr-1234.diff",
      "expected": "flag the missing null check in auth_service.go",
      "must_not_block": false
    },
    {
      "id": "pr-5678",
      "diff": "regression/diffs/pr-5678.diff",
      "expected": "no blocking comments",
      "must_not_block": true
    }
  ]
}

Step 3: Run the Suite on Every Configuration Change

#!/usr/bin/env bash
# review-regression.sh — run your AI reviewer against known PRs
set -euo pipefail

MANIFEST="regression/manifest.json"
REVIEW_CMD=${REVIEW_CMD:-"ai-review"}  # replace with your actual review command

for pr in $(jq -r '.[].id' "$MANIFEST"); do
  diff_file="regression/diffs/${pr}.diff"
  echo "=== ${pr} ==="
  $REVIEW_CMD --diff "$diff_file" > "regression/results/${pr}.txt" 2>&1
done

echo "Done. Inspect regression/results/*.txt against manifest expectations."

The script is deliberately simple because the value is in the manifest, not the automation. You can run it locally before merging a prompt change, or wire it into CI as a scheduled job. The important part is that results are written to files, so you can diff them across runs and see when review behavior changed.

Step 4: Commit Results and Review Them

Commit the manifest, the diffs, and the result files together. When someone changes the prompt or the model, they run the suite and commit the new results. This gives you a visible history of review behavior, and it makes the review configuration reviewable like your application code.

Why Free Model Access Changes This Workflow

A regression suite is a low-volume, continuous workload: ten diffs per configuration change, not thousands of tokens per developer per day. That is exactly the kind of workload where free model access and a free server option remove the last excuse for skipping the suite. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If every prompt tweak costs money, you will run the suite once and then stop. If it costs nothing, you can run it in CI on every pull request that touches your review configuration.

The free server option matters for a different reason: it keeps the suite out of your paid CI budget. A ten-diff run is small, but it still consumes compute, and teams that pay per minute will start skipping it on busy days. A free tier turns the regression suite into a habit instead of a cost decision, and habits are what actually protect your merge quality.

Limitations

Ten PRs is not a statistical sample; it is a smoke test. It will not catch every regression, and it can give false confidence if your team's bug patterns shift quickly. The suite also depends on historical judgment, so if your past reviews missed defects, the suite quietly encodes those misses as expected behavior.

Free model tiers may have rate limits or latency that make the suite slow, so measure the runtime before wiring it into CI. The suite tests the reviewer, not the review process; a reviewer that passes can still fail on a novel codebase or an unfamiliar framework. And the manifest needs maintenance, which is real work that teams often underestimate.

Who Should Not Use This Approach

Teams without a stable review history will find the suite stale quickly, so new repositories should build it from the first ten meaningful PRs. Teams that cannot commit to updating the manifest when bug patterns shift will watch the suite rot into a ritual. If your team reviews every PR manually anyway, the regression suite adds process without adding signal.

The Bottom Line

Benchmarks tell you what a model can do in general; a regression suite tells you what your reviewer does on your code. The latter is the only signal that matters when you merge, and it costs almost nothing to build. Free model access makes the suite a zero-cost habit, and that is the entire point: the cheapest verification is the one you actually run.

Opinion: Your AI Reviewer Is the Only Pipeline Stage Nobody Tests

Avery Lin — Tue, 25 Aug 2026 17:01:48 +0000

Every stage of your delivery pipeline has tests except the stage that judges the code itself. Your AI reviewer is the only component whose output is trusted without assertions, which makes it the largest unverified surface in modern CI. Pinning a prompt and a model version does not pin behavior, because upstream updates change outputs silently and nobody notices. The fix is a planted-bug corpus that measures reviewer drift on a schedule you can actually afford.

Drift Is Invisible Until a Bug Ships

Application code fails loudly when a test breaks, but review prompts fail quietly when a model update shifts their judgment. A reviewer that caught every null dereference in January can miss half of them in June while still producing confident, well-formatted comments. Teams notice only when a defect reaches production, and by then the review log looks identical to the one from the good months. Recent DEV discussions have asked who tests the reviewer, and the honest answer is almost nobody.

The root cause is that review output has no oracle. A linter has a rule set, a unit test has an assertion, and a human reviewer has a second human in code review. An AI reviewer has only the prompt, and the prompt is evaluated by a model that changes underneath it. If you cannot state what the reviewer must catch, you cannot detect when it stops catching it.

Pinning the Config Does Not Pin the Model

Pinning your review configuration to a specific model version is necessary but insufficient, because the identifier you pin is a moving target. Providers ship updates, adjust quantization, and alter default parameters without changing the version string you recorded. The configuration stays stable while the behavior underneath it drifts, which means your regression suite must assert on outputs rather than versions. Treat the reviewer like a dependency with behavioral contracts, not like a settings file.

This is the gap between the common advice to pin your AI review configuration and the reality of model deployment. Pinning gives you a reproducible starting point, but it gives you nothing to compare against when the model changes. What you actually need is a baseline of expected behavior and a cheap way to re-measure it on a regular cadence.

The Artifact: A Planted-Bug Review Corpus

The practical fix is a small corpus of synthetic pull requests, each containing one planted defect with a known bug class and an expected verdict. You build it once from your own incident history, then run the reviewer against it on a schedule. Here is a minimal corpus entry:

{
  "corpus_version": "2026-08",
  "min_acceptable_rate": 0.8,
  "cases": [
    {
      "id": "null-deref-01",
      "bug_class": "null_dereference",
      "should_catch": true,
      "file": "src/cache.js",
      "before": "const entry = cache.get(key);\nif (entry.expiresAt < Date.now()) {",
      "after": "const entry = cache.get(key) || null;\nif (entry.expiresAt < Date.now()) {"
    }
  ]
}

The before and after fields render as a synthetic diff, and should_catch records the verdict a competent reviewer must reach. Include clean cases too, because a reviewer that flags everything is as useless as one that flags nothing. The scoring script below is a minimal template; adapt the review() function to your own tool.

import json
import subprocess
import sys

def review(prompt: str) -> str:
    """Adapt this to your reviewer's CLI or API."""
    return subprocess.run(
        ["your-reviewer", "review"],
        input=prompt, capture_output=True, text=True
    ).stdout

def build_prompt(case: dict) -> str:
    return (
        f"Review this change to {case['file']}.\n"
        f"Before:\n{case['before']}\n"
        f"After:\n{case['after']}\n"
        "Report only concrete defects with file and line."
    )

def main(corpus_path: str) -> None:
    corpus = json.load(open(corpus_path))
    caught = 0
    for case in corpus["cases"]:
        output = review(build_prompt(case))
        flagged = case["id"] in output or case["bug_class"] in output
        caught += int(flagged)
        print(f"{case['id']}: {'caught' if flagged else 'MISSED'}")
    rate = caught / len(corpus["cases"])
    print(f"catch_rate={rate:.0%}")
    if rate < corpus["min_acceptable_rate"]:
        sys.exit(1)

if __name__ == "__main__":
    main(sys.argv[1])

The script exits nonzero when the catch rate drops below your threshold, so the regression run plugs directly into CI. The verdict check is intentionally naive, and a real implementation should parse the reviewer's structured output instead of searching for an ID string. The template contains no vendor-specific calls, because the corpus should outlive any single reviewer tool.

Why Free Infrastructure Changes the Habit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A regression suite only becomes a habit when its marginal cost is zero, because teams skip paid checks in the weeks when budgets are tight. MonkeyCode's free model access and free server option remove that excuse by making the weekly corpus run a background process rather than a purchasing decision. The economics matter because reviewer drift is slow, and a quarterly paid audit will miss the week when the catch rate quietly halves. A free weekly run turns drift detection into something you notice before the bug ships, not after.

The Workflow in Five Steps

Build a corpus of ten to twenty planted bugs from your recent incident history, one defect per case, plus a few clean diffs.
Run a baseline and record the catch rate for each bug class so you know what the reviewer actually detects today.
Schedule the corpus as a weekly cron job that fails when the catch rate falls below your threshold.
When it fails, diff the current review output against the baseline to decide whether the prompt or the model changed.
Prune the corpus quarterly, because bug classes your team no longer writes will inflate your confidence.

The fifth step matters more than it looks. A corpus that never changes measures history rather than current risk, and stale cases produce a catch rate that feels reassuring while the reviewer drifts on the bugs you actually ship. Maintenance is part of the artifact, not an afterthought.

Who Should Not Use This

Planted bugs are not real bugs, and a corpus of fifteen cases produces percentages with wide error bars, so treat the number as a trend indicator rather than a benchmark. Teams without an existing AI review step should not build this harness first, because measuring a reviewer you do not use is pure overhead. Small repositories with few weekly PRs will get more signal from a second human reviewer than from a drift detector. The corpus also catches failures of detection, not failures of judgment, which is a real limit of any output-based check.

If you already run an AI reviewer, the cheapest correctness test you can add this week is ten planted bugs and one cron job. The reviewer judges your code, and it is time something judged the reviewer.

Opinion: Per-Commit AI Review Is Impossible Until the Marginal Cost Is Zero

Avery Lin — Tue, 25 Aug 2026 06:27:07 +0000

AI review belongs on every commit, not just on every pull request, because the cheapest defect is the one caught in the session that wrote it. Metered pricing makes that cadence impossible, since each extra review adds a line-item cost that someone must justify. A free, always-on local server removes the marginal cost entirely, and that change alters review timing more than any model upgrade ever will.

The timing problem is really a pricing problem

Teams do not skip per-commit review because they doubt its value; they skip it because every call is a metered transaction. When each review costs money, the rational workflow is to batch diffs and review once at PR time. That batching is exactly what makes the feedback useless, because the author has already moved on to three other tasks. The real cost is not the dollar amount; it is the delay that the dollar amount justifies.

A free server changes the equation

MonkeyCode's free model access and free server option flip this equation by making the review endpoint a local resource instead of a metered API. Disclosure: This article was prepared as part of MonkeyCode's product outreach. When the server runs on a machine you control, one review costs the electricity of a single request, which is close enough to zero that nobody has to approve it. Review frequency stops being a budget decision and becomes a workflow decision.

The position here is deliberately narrow. Free models are not equal to frontier models on ambiguous architectural judgment, and pretending otherwise produces false confidence. The claim is that for the common class of concrete defects — null dereferences, resource leaks, inverted conditions — a free local reviewer on every commit beats a smarter reviewer on every tenth commit. This is also not the free-tier-in-CI argument, because a vendor free tier still sits behind a meter and a rate limit; a local server has neither.

The artifact: a pre-commit hook pointed at a local server

The pattern below is illustrative, and the exact endpoint and model string depend on the installed version of the server. Check the server's own help output before wiring anything to it, because flag names change between releases.

Step 1: Run the server on a machine that stays awake

A laptop that sleeps at 6 PM will silently turn your review pipeline off, so use a spare box, a NAS, or an always-on runner. Treat the server process like a database: start it once, supervise it, and notice when it dies. The machine only needs to be reachable from wherever you commit.

Step 2: Add a hook that reviews only the staged diff

#!/usr/bin/env bash
# .git/hooks/pre-commit — illustrative pattern; adapt to your server's API.
set -euo pipefail

DIFF="$(git diff --cached --unified=3)"
[[ -n "$DIFF" ]] || exit 0

PROMPT="You are a conservative reviewer. The text below is a staged diff.
Report only concrete defects: null dereferences, resource leaks, race
conditions, and inverted logic. Ignore style. If nothing is concrete,
reply with exactly NO_ISSUES.

$DIFF"

RESPONSE="$(curl -s http://127.0.0.1:8765/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d "$(jq -n --arg p "$PROMPT" \
        '{model:"local-free", messages:[{role:"user", content:$p}]}')")"

REVIEW="$(jq -r '.choices[0].message.content' <<<"$RESPONSE")"
printf '%s\n' "$REVIEW"

grep -q '^NO_ISSUES$' <<<"$REVIEW" || {
  echo "AI review flagged the staged diff; fix it or use --no-verify deliberately."
  exit 1
}

The hook sends only the staged diff, never the whole repository, which keeps the request small and the response specific. The output contract is strict: either the exact token NO_ISSUES or a list of concrete defects. A free model will occasionally invent style complaints, so the strict contract is what makes the hook automatable.

Step 3: Add a severity gate for the noise

A free model will sometimes flag a variable name or a comment style, and failing the commit on that noise trains developers to bypass the hook. Filter the response through a small set of high-signal tokens before deciding to fail:

grep -E 'race|leak|null|deadlock|use-after-free' <<<"$REVIEW" \
  && exit 1 || exit 0

The gate is crude but effective, and it encodes an honest limitation: the free reviewer is trusted for memory-safety and concurrency defects, not for taste. If the gate blocks a real defect that uses different wording, add that wording to the list.

Step 4: Let the hook force smaller commits

If the hook regularly sees 800-line diffs, the response degrades into generic advice, because the context window fills with noise. The hook therefore becomes a forcing function for smaller commits, which is a benefit that has nothing to do with AI. The constraint shapes the behavior, and the behavior is better engineering.

A decision table for when the free server is enough

Change type	Reviewer	Cadence
Small feature commit	Free local server	Every commit
Mechanical refactor	Free local server	Every commit
Large architectural PR	Paid model	Once, at PR time
Security-sensitive change	Paid model plus human	Once, at PR time
Generated code and lockfiles	None	Never

The table encodes the actual position: free local review is the default, and paid review is the exception that requires a reason. Most teams run the inverse, and that inversion is why their feedback arrives after the context is gone. The table also keeps the free server honest, because it does not claim to replace human judgment on the changes that matter most.

Limitations and who should not use this

The approach breaks down when diffs routinely exceed the local model's context window, because the reviewer then summarizes instead of inspecting. It also assumes an always-on machine, which rules out developers whose only hardware is a sleeping laptop. Regulated teams that need vendor audit trails and centralized review history should keep their paid pipeline, since logs on your own server are harder to present to a compliance officer. Solo developers and small teams with modest diff sizes are the audience, and anyone outside that audience should read this as an argument about timing, not as a universal setup guide.

The conclusion is about timing, not model quality

None of this claims that free models equal frontier models, and such a claim would be dishonest. The argument is that review timing beats review intelligence, because a mediocre review at commit time prevents a defect that a brilliant review at merge time merely documents. Try the hook on a side project for one week and count the defects caught before the commit; the count will decide whether your PR-time reviewer is a gate or a postmortem.