HTTP 2.4 AH00051: child pid 22471 exit signal Segmentation fault (11)

My companies website has been sporadically up and down lately; so was reported by various Directors. However all of our monitoring says uptime is great, Apache and MySQL are running. That’s great right? Wrong.

Come the dreaded intermittent downtime errors that have eluded many a system administrator. Often times most just shrug off this issue as a “network” issues meaning they don’t want to dive down and actually figure out what is causing the problem. In this case a simple look at the error log will be enough to sound the alarm.

Error log

[[email protected]_server httpd]# tail error_log
[Tue Feb 09 21:14:38.826237 2016] [core:notice] [pid 731] AH00052: child pid 16102 exit signal Segmentation fault (11)
[Tue Feb 09 21:19:36.150768 2016] [core:notice] [pid 731] AH00052: child pid 15050 exit signal Segmentation fault (11)
[Tue Feb 09 21:23:39.416821 2016] [core:notice] [pid 731] AH00052: child pid 16106 exit signal Segmentation fault (11)
[Tue Feb 09 21:52:47.209810 2016] [core:notice] [pid 731] AH00052: child pid 17844 exit signal Segmentation fault (11)
[Tue Feb 09 22:20:15.919584 2016] [core:notice] [pid 731] AH00052: child pid 17842 exit signal Segmentation fault (11)

So we know some child process is dying, but what now? First lets enable coredumping so we can actual have something to debug further.

Enable core dumps

ulimit -c unlimited

Add the following to your httpd.conf

# Enabling Core Dumps
CoreDumpDirectory /tmp

And restart apache

systemctl restart httpd

Alright now we will have something to actually start this debug process. So go to the problem site, and click all the buttons. Or write a quick spider to do it for you (that way if you ever have the problem again, you can just run your spider).

Now that we have coredumps enabled lets confirm we are actually getting core dumps.

[[email protected]_server ~]# tail -f /var/log/httpd/error_log
[Tue Feb 09 23:36:34.095785 2016] [:notice] [pid 22467] ModSecurity: LUA compiled version="Lua 5.1"
[Tue Feb 09 23:36:34.095786 2016] [:notice] [pid 22467] ModSecurity: LIBXML compiled version="2.9.1"
[Tue Feb 09 23:36:34.146866 2016] [auth_digest:notice] [pid 22467] AH01757: generating secret for digest authentication ...
[Tue Feb 09 23:36:34.147479 2016] [lbmethod_heartbeat:notice] [pid 22467] AH02282: No slotmem from mod_heartmonitor
[Tue Feb 09 23:36:34.148733 2016] [ssl:warn] [pid 22467] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Tue Feb 09 23:36:34.175972 2016] [mpm_prefork:notice] [pid 22467] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips PHP/5.4.16 configured -- resuming normal operations
[Tue Feb 09 23:36:34.175993 2016] [core:notice] [pid 22467] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Tue Feb 09 23:37:41.213259 2016] [core:notice] [pid 22467] AH00051: child pid 22471 exit signal Segmentation fault (11), possible coredump in /tmp
[Tue Feb 09 23:51:30.078761 2016] [core:notice] [pid 22467] AH00051: child pid 22480 exit signal Segmentation fault (11), possible coredump in /tmp
[Tue Feb 09 23:56:15.390730 2016] [core:notice] [pid 22467] AH00051: child pid 22473 exit signal Segmentation fault (11), possible coredump in /tmp

Check out a core dump and backtrace

Great we see possible coredump in /tmp, so lets check that out.

[[email protected]_server tmp]# find -mmin -1
./systemd-private-censor-httpd.service-mUXrIa/tmp
./systemd-private-censor-httpd.service-mUXrIa/tmp/core.22471

I didn’t have gdb on this server, so lets go ahead and install it just in case.

[[email protected]_server tmp]# cd ./systemd-private-censor-httpd.service-mUXrIa/tmp/
[[email protected]_server tmp]# gdb /usr/sbin/httpd core.22471
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/httpd...Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done.
(no debugging symbols found)...done.
[New LWP 22471]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/httpd -DFOREGROUND'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f9b1d753310 in fetch_timezone_offset () from /etc/httpd/modules/libphp5.so
Missing separate debuginfos, use: debuginfo-install httpd-2.4.6-40.el7.x86_64
(gdb) bt full
#0  0x00007f9b1d753310 in fetch_timezone_offset () from /etc/httpd/modules/libphp5.so
#1  0x00007f9b1d754be4 in timelib_get_time_zone_info () from /etc/httpd/modules/libphp5.so
#2  0x00007f9b1d72c2a0 in date_format () from /etc/httpd/modules/libphp5.so
#3  0x00007f9b1d72dea0 in zif_date_format () from /etc/httpd/modules/libphp5.so
#4  0x00007f9b1d976f8c in zend_do_fcall_common_helper_SPEC () from /etc/httpd/modules/libphp5.so
#5  0x00007f9b1d8f4077 in execute () from /etc/httpd/modules/libphp5.so
#6  0x00007f9b1d8bdac0 in zend_call_function () from /etc/httpd/modules/libphp5.so
#7  0x00007f9b1d8048e3 in zif_call_user_func_array () from /etc/httpd/modules/libphp5.so
#8  0x00007f9b1d976f8c in zend_do_fcall_common_helper_SPEC () from /etc/httpd/modules/libphp5.so
#9  0x00007f9b1d8f4077 in execute () from /etc/httpd/modules/libphp5.so
#10 0x00007f9b1d8bdac0 in zend_call_function () from /etc/httpd/modules/libphp5.so
#11 0x00007f9b1d8048e3 in zif_call_user_func_array () from /etc/httpd/modules/libphp5.so
#12 0x00007f9b1d976f8c in zend_do_fcall_common_helper_SPEC () from /etc/httpd/modules/libphp5.so
#13 0x00007f9b1d8f4077 in execute () from /etc/httpd/modules/libphp5.so
#14 0x00007f9b1d8ccccf in zend_execute_scripts () from /etc/httpd/modules/libphp5.so
#15 0x00007f9b1d86c6e6 in php_execute_script () from /etc/httpd/modules/libphp5.so
#16 0x00007f9b1d978c8d in php_handler () from /etc/httpd/modules/libphp5.so
#17 0x00007f9b2e877240 in ap_run_handler ()
#18 0x00007f9b2e877789 in ap_invoke_handler ()
#19 0x00007f9b2e88b5cc in ap_internal_redirect ()
#20 0x00007f9b26e09dec in handler_redirect () from /etc/httpd/modules/mod_rewrite.so
#21 0x00007f9b2e877240 in ap_run_handler ()
#22 0x00007f9b2e877789 in ap_invoke_handler ()
#23 0x00007f9b2e88bb0a in ap_process_async_request ()
#24 0x00007f9b2e88bde4 in ap_process_request ()
#25 0x00007f9b2e888772 in ap_process_http_connection ()
#26 0x00007f9b2e880810 in ap_run_process_connection ()
#27 0x00007f9b2425280f in child_main () from /etc/httpd/modules/mod_mpm_prefork.so

I had initially assumed that the issue was PHP because… well… it’s PHP. But this actually confirms my theory. (Remember troubleshooting is all about the scientific method)

So with this new found trace I know that the fetch_timezone_offset() method and/or function (depending who you’re talking to dev vs admin) is the last thing called before dying. Now time for some GoogleFu, I searched for: “php segfault 11 in fetch_timezone_offset” and got the following bug:
https://bugs.php.net/bug.php?id=66721

Now this was the first bug that came up so lets not blindly assume this is the issue. Lets test if this is the issue with the test script provided:

[[email protected]_server ~]# cat test.php
<?php
$y = 'O:8:"DateTime":3:{s:4:"date";s:19:"2014-02-15 02:00:51";s:13:"timezone_type";i:3;s:8:"timezone";s:10:"1234567890";}';

var_dump(unserialize($y)); // segfault
?>
[[email protected]_server ~]# php test.php
Segmentation fault (core dumped)

So we know that this test script fails but this doesn’t really narrow down the problem to this exact bug we need to run at a minimum one more test to ensure that all of PHP isn’t broken, lets do a simple phpinfo test:

[[email protected]_server ~]# cat test2.php
<?php
phpinfo();
?>
[[email protected]_server ~]# php test2.php | head -2
phpinfo()
PHP Version => 5.4.16

Great so with this narrowed down we can safely assume that the issue is the same as the one reported in the bug report. So lets get to patching PHP which will hopefully lead to me sleeping.

Patch / Upgrade / Fix

So the simplest fix is to just upgrade PHP, so that’s what we’re going to do in this situation, mainly because a Patch wasn’t attached to the bug.

wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-7.rpm
rpm -Uvh remi-release-7*.rpm epel-release-7*.rpm
yum -y install php56 # plus whatever modules you need
systemctl restart httpd

After this you need to try and recreate the issue, first by testing the test script.

[[email protected]_server ~]# cat test.php
<?php
$y = 'O:8:"DateTime":3:{s:4:"date";s:19:"2014-02-15 02:00:51";s:13:"timezone_type";i:3;s:8:"timezone";s:10:"1234567890";}';

var_dump(unserialize($y)); // segfault
?>
[[email protected]_server ~]# php test.php
PHP Fatal error:  Invalid serialization data for DateTime object in /root/test.php on line 4
PHP Stack trace:
PHP   1. {main}() /root/test.php:0
PHP   2. unserialize() /root/test.php:4
PHP   3. DateTime->__wakeup() /root/test.php:4

Alright, finally getting some PHP errors vs it just dying. That’s what we are looking for. Also make sure you crawl your website again. And hopefully after your crawl your log looks like this:

[[email protected]_server ~]# tail -f /var/log/httpd/error_log
[Wed Feb 10 00:41:04.993265 2016] [:notice] [pid 27107] ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/) configured.
[Wed Feb 10 00:41:04.993270 2016] [:notice] [pid 27107] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"
[Wed Feb 10 00:41:04.993283 2016] [:notice] [pid 27107] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
[Wed Feb 10 00:41:04.993286 2016] [:notice] [pid 27107] ModSecurity: LUA compiled version="Lua 5.1"
[Wed Feb 10 00:41:04.993288 2016] [:notice] [pid 27107] ModSecurity: LIBXML compiled version="2.9.1"
[Wed Feb 10 00:41:05.044647 2016] [auth_digest:notice] [pid 27107] AH01757: generating secret for digest authentication ...
[Wed Feb 10 00:41:05.309845 2016] [mpm_prefork:notice] [pid 27107] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips PHP/5.6.18 configured -- resuming normal operations
[Wed Feb 10 00:41:05.309879 2016] [core:notice] [pid 27107] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

Write a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.