Saturday, July 14, 2007

[BLUG] Running out of memory during log turnover

Hello,

I'm wondering if anybody knows what I might do to address this problem.
I'm using FreeBSD and Apache is having a hard time when my Apache error
log files turn over causing Apache to crash with the following output to
my error log... No other logs of any sort on the system cause this
problem, but this happens consistently with my Apache error log:

Jul 14 23:00:00 netmusician newsyslog[81819]: logfile turned over due to
size>5000K
Out of memory during request for 32 bytes, total sbrk() is 10196992 bytes!
Callback called exit at
/usr/local/lib/perl5/site_perl/5.8.8/HTML/Element.pm line 1502.
BEGIN failed--compilation aborted at
/usr/local/lib/perl5/site_perl/5.8.8/HTML/Formatter.pm line 106.
Callback called exit at
/usr/local/lib/perl5/site_perl/5.8.8/HTML/Formatter.pm line 106.
BEGIN failed--compilation aborted at
/usr/local/lib/perl5/site_perl/5.8.8/HTML/FormatText.pm line 68.
Callback called exit at
/usr/local/lib/perl5/site_perl/5.8.8/HTML/FormatText.pm line 68.
BEGIN failed--compilation aborted at
/usr/local/rt3/lib/RT/Transaction_Overlay.pm line 84.
Callback called exit at /usr/local/rt3/lib/RT/Transaction_Overlay.pm
line 84.
BEGIN failed--compilation aborted at
/usr/local/rt3/lib/RT/Transactions.pm line 73.
Callback called exit at /usr/local/rt3/lib/RT/Transactions.pm line 73.
BEGIN failed--compilation aborted at /usr/local/rt3/lib/RT/Record.pm
line 1490.
Callback called exit at /usr/local/rt3/lib/RT/Record.pm line 1490.
BEGIN failed--compilation aborted at
/usr/local/rt3/lib/RT/CurrentUser.pm line 72.
Callback called exit at /usr/local/rt3/lib/RT/CurrentUser.pm line 72.
BEGIN failed--compilation aborted at /usr/local/rt3/lib/RT.pm line 49.
Callback called exit at /usr/local/rt3/lib/RT.pm line 49.
BEGIN failed--compilation aborted at /usr/local/rt3/bin/webmux.pl line 66.
Callback called exit at /usr/local/rt3/bin/webmux.pl line 66.

It looks like RT is a part of this problem consistently each time this
has happened... RT is setup to use mod_perl.

I've tried various Apache tuning techniques thinking that there was some
sort of memory leak, but I haven't been able to isolate this problem.
Since it is a PITA to manually have to restart Apache every time this
happens, I'm hoping that some of you may have some ideas as to some
things I can try here?

The other reason why I think this might be an Apache problem is because
of the several constant error messages that I see in my error log such as:

[Sat Jul 14 23:11:12 2007] [info] Subsequent (No.2) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:12 2007] [info] Subsequent (No.3) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:12 2007] [info] Subsequent (No.4) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:12 2007] [info] Subsequent (No.5) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:12 2007] [info] Subsequent (No.6) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:13 2007] [info] Subsequent (No.7) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:13 2007] [info] Subsequent (No.8) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.9) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.10) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.11) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Connection to child 3 established
(server netmusician.org:443, client 74.130.30.11)
[Sat Jul 14 23:11:14 2007] [info] Seeding PRNG with 136 bytes of entropy
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.12) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Initial (No.1) HTTPS request received
for child 3 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.13) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.2) HTTPS request
received for child 3 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.14) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:15 2007] [info] Subsequent (No.3) HTTPS request
received for child 3 (server netmusician.org:443)
[Sat Jul 14 23:11:15 2007] [info] Subsequent (No.15) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:15 2007] [info] Subsequent (No.4) HTTPS request
received for child 3 (server netmusician.org:443)
[Sat Jul 14 23:11:30 2007] [info] (70007)The timeout specified has
expired: SSL input filter read failed.
[Sat Jul 14 23:11:30 2007] [info] Connection to child 2 closed with
standard shutdown(server netmusician.org:443, client 74.130.30.11)
[Sat Jul 14 23:11:30 2007] [info] (70007)The timeout specified has
expired: SSL input filter read failed.
[Sat Jul 14 23:11:30 2007] [info] Connection to child 3 closed with
standard shutdown(server netmusician.org:443, client 74.130.30.11)

I'm not entirely convinced that my Apache install is healthy, although
it seems to behave just fine and I've noticed no real problems with
Apache. It has been perfectly stable, outside of this reproducible crash
caused by log file turnover.

I'm kind of grasping at straws here, I know.


--
Joe Auty
NetMusician: web publishing software for musicians
http://www.netmusician.org
joe@netmusician.org
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug