Your Cron Job Is Lying to You: A Dependency-Free Watchdog for Silent Failures

A green checkmark in your cron logs means nothing. Here is the three-layer defense that turns "the process exited 0" into "the job actually did its work" — with zero extra dependencies.

You set up a nightly sync job, a cleanup worker, or a scraper. The logs say "completed." Your monitoring dashboard is green. Three weeks later you discover the database has been quietly missing records the whole time, or a cloud box ran for days instead of minutes and quietly burned your credits.

This is the single most common way unattended jobs fail, and it is also the most embarrassing, because nothing looks broken. In August 2026 alone, several DEV.to authors independently reported the exact same shape of bug: a scheduled task that reported success every 5 minutes while the process inside it had been crashing for weeks (search "The Exit Code That Lied" or "My scheduled task reported success every 5 minutes" if you want company).

The trap is not one bug — it is three layers of failure that reinforce each other. Fix all three, and your cron jobs stop lying. None of the fixes require a new service, a paid monitor, or anything beyond bash and Python that's already on your box.

The three layers of the silent-failure trap

Most "fixes" only patch one layer. Let's patch all three.

Layer 1 — Stop the pipeline from lying about the exit code

The classic footgun:
{% raw %}

python3 sync.py 2>&1 | tee -a sync.log
echo "exit: $?"   # always 0, even when sync.py crashed

$? here is the exit status of tee, not sync.py. The shell keeps the real per-command statuses in the PIPESTATUS array, but almost nobody reads it.

The one-line fix is set -o pipefail in the wrapper script (documented in the GNU Bash manual: with pipefail, "the pipeline's return status is the value of the last command to exit with a non-zero status, or zero if all commands exit successfully"). Combined with set -e, a failure anywhere in the pipeline aborts the script with a non-zero code.

#!/usr/bin/env bash
# run_guarded.sh — run a job and never lie about its outcome
set -euo pipefail

JOB_NAME="${1:?usage: run_guarded.sh <job-name> <command...>}"
shift

LOG_DIR="${LOG_DIR:-/var/log/jobs}"
mkdir -p "$LOG_DIR"
LOG="$LOG_DIR/$JOB_NAME.log"

# Run the real command. PYTHONUNBUFFERED=1 forces unbuffered output from any
# Python the job spawns (Layer 2). pipefail makes the pipeline fail if the job
# fails; set -e propagates that non-zero exit out of this wrapper.
PYTHONUNBUFFERED=1 "$@" 2>&1 | tee -a "$LOG"

That single set -euo pipefail line catches Layer 1. set -u (treat unset variables as errors) and set -o errexit are documented bash options; together they make the script fail loud instead of failing silent.

Common mistake: putting set -e at the top but then wrapping the body in if ...; then or cmd || true, which defeats it. Don't swallow the failure you just armed the script to report.

Layer 2 — Kill the buffering so you can see why it died

Python block-buffers stdout when it is not attached to a terminal (confirmed in the Python docs: "Otherwise, it is block-buffered like regular text files"). So a crash that happens right after a print("about to process X") loses that line, because it never reached the file before the process was reaped.

Two equivalent fixes:

python3 -u sync.py            # -u: force stdout/stderr unbuffered
# or:
PYTHONUNBUFFERED=1 python3 sync.py

The Python docs state PYTHONUNBUFFERED "set to a non-empty string ... is equivalent to specifying the -u option." Either way, the last thing your job printed is now in the log, exactly where the crash happened.

Common mistake: relying on logging with default config. The logging module writes to stderr and is usually line-buffered, but if you replace sys.stdout or pipe through another buffer, you can reintroduce the problem. When in doubt, set PYTHONUNBUFFERED=1 at the wrapper level so it covers everything the job spawns.

Layer 3 — "Exited 0" is not "did its job"

This is the layer most monitoring misses. A dead-man's-switch (a service that pings "I'm alive every N minutes or alert") only proves the process is alive. It cannot tell you the job did nothing — e.g. an auth error that's caught, logged once, and loops forever producing zero real work while the process hums along happily.

You need a progress signal that is separate from the exit code. The cheapest one is a watermark: record the furthest point the job actually processed, and assert it advanced.

#!/usr/bin/env python3
# watermark_check.py — assert the job actually made progress
import os, sys, time

WATERMARK = "/var/run/job-status/sync.watermark"
MAX_STALE_S = 3600  # progress must advance at least hourly

def last_processed() -> float:
    try:
        with open(WATERMARK) as fh:
            return float(fh.read().strip())
    except FileNotFoundError:
        return 0.0

def main() -> int:
    now = time.time()
    age = now - last_processed()
    if age > MAX_STALE_S:
        # One positive marker string. A retry loop can grep for "not done:"
        # to tell "job needs work" apart from "checker itself is broken".
        print(f"not done: watermark stale ({age:.0f}s since last progress)", file=sys.stderr)
        return 1
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

The job records progress atomically so a crash mid-write can't leave a half-written watermark:

def record_progress(ts: float) -> None:
    tmp = WATERMARK + ".tmp"
    with open(tmp, "w") as fh:
        fh.write(str(ts))
    os.replace(tmp, WATERMARK)  # atomic rename, never leaves a corrupt file

Wire it into cron so the watermark check runs after the job:

# m h  dom mon dow   command
*/5  *  *   *   *   /opt/jobs/run_guarded.sh sync /opt/jobs/sync.py
*/5  *  *   *   *   /opt/jobs/watermark_check.py || curl -fsS https://hooks.example.com/alert/sync-failed

Now a job that runs, does nothing, and exits 0 still gets caught — because the watermark didn't move.

The heartbeat: catch a job that died without exiting

Layers 1–3 assume the process at least finishes. But on a free/tiny server, a process can be OOM-killed or reclaimed mid-run and leave no trace at all. That's where a heartbeat (dead-man's-switch on the work, not just the process) earns its keep.

#!/usr/bin/env bash
# check_heartbeat.sh — run from cron; alert if the job stopped touching its heartbeat
set -euo pipefail
JOB_NAME="${1:?usage: check_heartbeat.sh <job-name> [max_age_s]}"
MAX_AGE_S="${2:-300}"   # default: must tick within 5 minutes
HB="/var/run/job-health/$JOB_NAME.heartbeat"

now=$(date +%s)
last=$(stat -c %Y "$HB" 2>/dev/null || echo 0)
age=$(( now - last ))

if (( age > MAX_AGE_S )); then
  echo "ALERT: $JOB_NAME heartbeat stale (${age}s > ${MAX_AGE_S}s)" >&2
  exit 2   # distinct code: the *monitor* failed, not the job
fi
exit 0

The job touches $HB after every batch:

date +%s > /var/run/job-health/sync.heartbeat

Use exit code 2 for "the monitor failed" deliberately. If a retry loop wraps this checker, it must distinguish "job not done, retry it" (a not done: marker, exit 1) from "checker itself is broken" (exit 2) — otherwise you'll retry a job that was never actually broken, or page nobody when the checker is the thing that died. Match the marker, not the bare non-zero.

Putting it together: the three-layer scorecard

No single layer is sufficient. Together they cover every shape of silent failure the community has been hitting.

Caveats and trade-offs

• pipefail is bash-only. If your wrapper is #!/bin/sh on Debian/Ubuntu, /bin/sh is dash and silently ignores set -o pipefail. Always use #!/usr/bin/env bash for these wrappers.
• Don't set -e inside a function that's used in a condition. if myfunc; then disables errexit for the call; that's fine, but don't assume the function aborted the script.
• Watermarks need a real "did work" signal. For a sync job it's the newest upstream timestamp; for a scraper it's the highest item id; for a backup it's the size/checksum of the newest archive. Pick the metric that actually means "progress" — a heartbeat file alone does not.
• Alerts need a sink. The curl to a webhook above is a placeholder. Point it at whatever you already watch: email, a Slack/Discord incoming webhook, or even a second cron job that flips a file your existing monitor polls.
• This is monitoring, not magic. It tells you something is wrong; it won't auto-heal. Pair it with idempotent jobs (re-run safely after a crash) and you get fast detection + safe recovery.

Practical takeaways

1. Wrap every unattended job in set -euo pipefail and run Python with -u / PYTHONUNBUFFERED=1. That's Layer 1 + 2 in two lines.
2. Add a progress watermark and assert it advanced. "Exited 0" is not proof of work.
3. Add a heartbeat for anything that runs longer than a few minutes or lives on reclaimable infrastructure.
4. Give your checker a distinct exit code / marker so a retry loop can tell "job needs work" from "the checker is broken."
5. Make jobs idempotent so a re-run after a detected failure is safe.

Conclusion

Your cron logs were never the source of truth — they were a story the shell told itself. The fix isn't a new observability platform; it's three cheap, dependency-free guards: preserve the real exit code, unbundle the output, and prove the job actually progressed. Do those and the next time a job quietly stops working, you find out in five minutes instead of three weeks.

References

• GNU Bash Manual — Pipelines, pipefail, and PIPESTATUS: https://www.gnu.org/software/bash/manual/bash.html
• Bash FAQ 002 — capturing pipeline exit status: https://mywiki.wooledge.org/BashFAQ/002
• Python docs — -u / PYTHONUNBUFFERED and stream buffering: https://docs.python.org/3/using/cmdline.html
• Python sys docs — stdout/stderr buffering behavior: https://docs.python.org/3/library/sys.html
• Related community reports (the bug class this guards against), DEV.to, Aug 2026:
  • "The Exit Code That Lied: Debugging a Silent Failure on a Free Server"
  • "My scheduled task reported 'success' every 5 minutes for 3 weeks..."
  • "How a Silent Python TypeError Left Our Cloud Worker Running for 3 Days"