Jul 7th, 2012
Random CPU overload and freezes - Leap second
Yesterday, we have found an insane cpu overload generated from nowhere on several machines we manage, this was related to The leap second that was inserted on Saturday night, this can cause permanent high CPU loads on Linux computers (Debian and CentOS/RHEL mainly). Among other examples, the behaviour is documented in the blog of the Mozilla Foundation, where strange peak loads on one Mozilla server were observed from the time the leap second was added.
A quick fix to solve this is by performig a quick reboot of the server (Notice that rebooting server services such as httpd, mysqld, even tomcat will not work). If you found a reboot it is impossible because of downtime issues, you can solve the issue by stopping the NTP daemon and executing a Perl script that reset the leap second bit in the kernel, the perl script can be found on the following article,
http://my.opera.com/marcomarongiu/blog/2012/06/01/an-humble-attempt-to-work-around-the-leap-second
The root of the behaviour is located in the Linux kernel, where a leap second that is triggered by the NTP subsystem results in a deadlock situation. The problem appears to affect all kernel versions from 2.6.26 up to and including 3.3.
A further topic discussion concerning this behaviour can be followed on the link below,
Should you require further support to patch/upgrade your kernel or solve this issue, please submit a Hour of Support,
