What's the Most Reliable Way to Run Jobs Every Minute?

The Problem with Periodic Tasks

Automating tasks to run every minute is essential for many systems, whether it’s processing real-time data, managing logs, or syncing files. But achieving reliability and precision is trickier than it seems.

The classic cron job executes commands at fixed intervals but offers no insight into what happens when a job fails or takes longer than expected. Modern systemd timers improve on this with better logging and dependency handling, but they too can falter under certain conditions.

For projects requiring fine-grained control and resilience, I turned to a custom PHP daemon. It’s tailored for my needs and has proven robust in handling edge cases that traditional tools struggle with. In this post, I’ll explain the trade-offs of these approaches and share details about my implementation.

Common Issues in Periodic Task Scheduling

Before diving into the solutions, let’s examine the potential pitfalls that can disrupt periodic jobs:

1. Overlapping Executions:
   When a job takes longer than its interval (e.g., a 70-second task scheduled every 60 seconds), subsequent executions may overlap or be skipped entirely.

2. Missed Schedules:
   Cron jobs and systemd timers don’t monitor whether tasks actually run successfully. If a job fails or the system is under heavy load, it might not execute at all.

3. Lack of Feedback:
   With cron, there’s no built-in mechanism to log errors or notify you if something goes wrong. Systemd improves this, but analyzing logs can still be cumbersome.

4. Startup Delays:
   After a reboot, timers may not execute immediately, leaving gaps in your task schedule.

5. Resource Utilization:
   Poorly designed loops or overlapping tasks can consume unnecessary CPU and memory, leading to system instability.

Solution 1: Systemd Timers—Reliable, but with Caveats

Systemd timers are a powerful and modern alternative to cron. They handle dependencies and logging better, making them ideal for complex workflows.

Potential Issue: Overlapping Tasks

By default, systemd timers don’t manage task execution time. If a task exceeds its interval, subsequent executions are skipped.

Example: A task scheduled to run every minute that takes 70 seconds to complete:

[Timer]
OnCalendar=*-*-* *:00
Persistent=true

Here’s what happens:

1. The first execution starts at 00:00 but runs until 00:01:10.
2. The next execution is skipped because the task is still running.

Solution: Use RemainAfterExit or ExecStartPre to manage long-running tasks, or switch to a custom solution like a PHP daemon to ensure granular control. For small side-projects which have short running tasks this is working very well, but for bigger projects and long running cron jobs I wasn't able to work with systemd reliably and switched to my custom daemon.

Strengths of Systemd Timers

• Robust logging and monitoring with journalctl.
• Integration with service dependencies (e.g., only run a task if the database is active).
• Resilient to reboots with Persistent=true.

Solution 2: Cron—Classic and Lightweight

Cron is the go-to for simple periodic tasks. Its configuration is easy to learn, and it’s available on almost every UNIX-like system.

Potential Issue: “Stupid Execution”

Cron blindly executes tasks based on the schedule without monitoring their success. If a job overlaps, takes longer than expected, or fails, cron won’t handle it.

Example: A database backup script scheduled to run every minute:

* * * * * /path/to/backup.sh

1. If the script takes 3 minutes, it will overlap with subsequent runs.
2. If the script fails due to disk space issues, there’s no notification or retry mechanism.

Strengths of Cron

• Simplicity: One line in a crontab does the job.
• Low resource usage: Cron wakes up only when needed.

The classic cron system works very well, when you just need a periodic command executor, where it isn't a big deal when it runs concurrent or misses one or two runs.

Solution 3: Custom PHP Daemon—Fine-Grained Control

For use cases where systemd and cron fall short, I created a custom PHP daemon. This approach gives me full control over scheduling, logging, and error handling. I use this daemon for my charm-based projects since a few months and it works perfectly, gives me control and is programmed fully in PHP. Of course you can create such daemons in other languages as well, but since this daemon is for a PHP framework, this is the way to go.

How My PHP Daemon Works

The daemon runs in a continuous loop, checking the system clock to execute tasks exactly at the start of each minute. It handles failures gracefully and ensures tasks don’t overlap. It just needs the "pcntl" PHP extension.

Here’s a simplified version:

class CronDaemon {
    private $running = true;

    public function run() {
        pcntl_signal(SIGTERM, [$this, 'signalHandler']);
        pcntl_signal(SIGINT, [$this, 'signalHandler']);

        while ($this->running) {
            $start_time = time();
            if (date('s', $start_time) === '00') {
                $this->executeTask();
                sleep(1); // Prevent double-execution within the same second
            }

            // Wait until the next minute
            do {
                usleep(500000); // Sleep for 0.5 seconds
                pcntl_signal_dispatch();
            } while (date('s', time()) !== '00');
        }
    }

    private function executeTask() {
        echo "[" . date('H:i:s') . "] Running task...\n";
        // Add your task logic here
    }

    public function signalHandler($signal) {
        $this->running = false;
    }
}

As said, I created such a daemon for my PHP framework. You can check out the detailed cron daemon code in the github repository: Charm Repository > Crown module > Console Jobs > CronDaemon.

Why It Works for Me

1. Granular Control:
   The loop ensures tasks are executed at precise intervals, even accounting for task duration.

2. Integration with Systemd:
   The daemon is wrapped in a systemd service to handle restarts on failure:

   [Service]
   ExecStart=/usr/bin/php /path/to/cron-daemon.php
   Restart=always

3. Error Handling:
   Tasks can be monitored directly within the PHP script, allowing for retries or notifications. Like a typical daemon, this one also saved the process ID (PID) of the daemon's main process in a cron_daemon.lock file, so the process can also easily be found in the system and also easily be displayed.

Potential Challenges

• Resource Usage:
  A running loop consumes more CPU than a passive cron job or timer. However, with efficient coding and usleep, this can be minimized. Every daemon needs some kind of loop / timer and I found the resource usage very low for my daemon. With PHP8.3 on my debian host it's not using the CPU at all while sleeping (0,0% CPU) and only needs 22MB of physical memory (0,1% MEM).

• Monitoring the Daemon:
  While systemd ensures restarts, additional logging and alerting mechanisms should be implemented to track the daemon’s health. Which is easily done, since logging can easily be added to the PHP script to the internal log handler or output it to STDOUT and STDERR so systemd can store it and we can view it via journalctl.

Conclusion: Why I Built My Own Daemon

While cron and systemd timers are excellent tools, they have limitations in handling overlapping tasks, detailed monitoring, and failure recovery. For scenarios where precision and reliability are non-negotiable, my custom PHP daemon, combined with systemd for management, provides the flexibility I need.

If you’re managing a system with similar requirements, consider writing a custom solution tailored to your needs—or reach out to discuss how I can help you build one!