Incident-Report: Patroni Failure Due to Time Travel

Ruohang FengPigsty founder, PostgreSQL mage, database guru, cloud-exit evangelist

Summary: Machine restarted due to failure, NTP service corrected PG time after PG startup, causing Patroni to fail to start.

The failure information in Patroni is shown as follows:

Process %s is not postmaster, too much difference between PID file start time %s and process start time %s

When patroni process start time and pid time are inconsistent, it assumes: postgres is not running.

If the two times differ by more than 30 seconds, patroni fails and cannot start.

The code that prints the error message is:

start_time = int(self._postmaster_pid.get('start_time', 0))
if start_time and abs(self.create_time() - start_time) > 3:
    logger.info('Process %s is not postmaster, too much difference between PID file start time %s and process start time %s', self.pid, self.create_time(), start_time)

Also discovered a BUG in Patroni: https://github.com/zalando/patroni/issues/811 The two timestamps in the error message are reversed.

Lessons learned: NTP time synchronization is very important