Engineering Case Study
Building Reliability Into Scheduled Systems
WordPress's built-in cron system works until it doesn't — and when it stops, it usually stops quietly.
The setup is common on hosting environments without access to true server-level cron: scheduled imports run through WP-Cron instead, pulling listing data from external APIs (RESO, YourMembership, and similar) into WordPress on a recurring basis. WP-Cron's trigger model means a scheduled task fires on page load rather than on a true system clock, which already introduces some imprecision. The bigger problem is rarer but more damaging: WP-Cron will occasionally drop a recurring event from its schedule entirely, silently, with no log entry indicating anything happened. The job just stops running.
The data stops updating. Nothing breaks loudly. The failure surfaces days later as a support ticket about stale listings.
The architecture that prevents this from becoming a data integrity problem has two layers.
Ingestion is separated from promotion.
API pulls write into temp tables, not directly into live tables. The import process paginates through the API's results until it signals no more pages remain, then sets a completion flag in the database. A separate cron job watches for that flag and handles moving temp data into live tables. This separation means a partial or interrupted pull — from API throttling, a timeout, a dropped connection — never corrupts what's actually serving the site.A minimum record count gate protects the promotion step.
Before the temp-to-live swap executes, the record count is checked against a minimum threshold. If an API gets throttled mid-pull and returns a fraction of the expected records, the swap simply doesn't happen. The live tables stay intact rather than getting overwritten with an incomplete dataset. This is a small check, but it's the difference between a quiet retry next cycle and silently serving broken data to users.The cron monitoring plugin closes the remaining gap.
It checks whether the specific named jobs that should exist in the WP-Cron schedule actually do. When one is missing — which happens more often than it should, for reasons that aren't always clear — the plugin recreates the schedule entry automatically, restoring the recurring pull without manual intervention.None of these are complicated fixes individually. What makes the system reliable is that they address failure at three different points: data quality before promotion, schedule integrity at the source, and silent failure detection across both. A system that fails loudly is frustrating to deal with. A system that fails silently is dangerous, because nothing prompts anyone to look until the damage is already visible.
Council Perspectives
The council reviewed the problem from five different angles. My responses add the operational context behind each observation.
Julie: For a while, I kept recurring calendar reminders to verify that every cron job still existed. Eventually I changed the architecture so the system could detect a missing schedule and repair itself.
Julie: Some of these API pulls were large enough that confirming a failure could take hours. You had to distinguish between a slow, paginated import and one that had quietly stopped altogether.
Julie: The plugin's dropped schedules were a known risk, and pagination multiplied the impact because each dataset depended on many separate pulls completing in sequence.
Julie: Often our clients discovered it only after their own users noticed stale information. By then, a quiet infrastructure failure had already become a visible trust problem.
Julie: Exactly. No warning, no useful error, and no dramatic failure. It simply disappears from the schedule.