Re: [BLUG] Fwd: possible causes of segfaults

Hiya,

Well, I'm using Ubuntu, and generally upgrades go fine without
rebooting... It's a nice server machine with ECC memory. So yeah, it
should really be at least telling me if I'm getting memory corruption,
if not fixing it outright.

Hmm, once, I had a computer that, on one version of Linux, was fine,
and on the next version (2.6.15->2.6.16 or something) would flip one
bit every 600MB of i/o. It slowly corrupted the entire filesystem and
... yuck. So I guess right now I'm suspecting Linux, or something
subtle in the hardware.

Thanks for your advice! It's good to narrow it down a bit. I'll
probably come in sometime over break (my boss will probably work the
whole week, but nobody else is here, and he can deal) and run that
memory test, as you say.

Bleh,
-Thomas

On Mon, Dec 21, 2009 at 4:33 PM, Steven Black <blacks@indiana.edu> wrote:
> Did you upgrade the system without rebooting it? (That should be
> reproducible, though.) Are you using a version of ImageMagick compiled
> for a different distribution of Linux? (I know a lot of RPM-based
> systems do not bundle a lot of programs.)
>
> If you downloaded an RPM that wasn't compiled specifically for your
> distribution/version, I would expect that to be the cause of the
> problem. If that's the cause of the problem, building it from source
> should clear it up.
>
> If you're using the version of ImageMagick that comes with your
> distribution and you've not recently performed an upgrade, memory
> corruption seems the most likely candidate. Does that machine have
> unparitied memory? Any type of memory other than unparitied would likely
> show an error instead of just producing bogus data. (It is why I hate
> unparitied memory.)
>
> It could also be a CPU fault.
>
> Problems with hard drives tend to show up as errors with the specific
> media. (It'll list the device producing the error.) Hardware problems
> regarding media do not normally produce segfaults, unless the
> application fails to handle the error case.
>
> My recommendation: Pick an upcoming weekend and tell them the services
> of this machine will be unavailable. Then start the memory test at
> 5:15pm (adjusted for the end of your workday) and run a memory checker
> all weekend. Then come in 15 minutes early on Monday to check for errors
> and reboot the system.
>
> Cheers,
> Steven Black
>
>
>
> On Mon, Dec 21, 2009 at 03:58:40PM -0500, Thomas Smith wrote:
>> Hiya,
>>
>> I just installed a new file / computation server at my work, and a few
>> days into its tenure, I've started noticing some segfaults.
>>
>> For instance, I was running a bunch of image processing jobs, and
>> ImageMagick's "convert" program segfaulted.  I ran it again on the
>> same data, and it did fine.  So I'm wondering if I have bad hardware,
>> or bad libraries, or what?
>>
>> Possible causes of segmentation faults that I know of:
>> Hardware --- could be very random and difficult to find, might need to
>> totally shut down the server and run a memory tester for days in order
>> to find.
>>
>> Filesystem corruption --- should be reproducible, right?  if "convert"
>> segfaults once, it should do it again...
>>
>> Libraries / os problems --- should be reproducible too, right?
>>
>>
>>
>> from dmesg:
>> [17217.872070] ld[16027]: segfault at 0 ip 00002ae3b445411b sp
>> 00007fffc39a74e8 error 4 in libc-2.9.so[2ae3b43d0000+168000]
>> [17354.753195] ld[20115]: segfault at 0 ip 00002b832843d11b sp
>> 00007fff729f2fa8 error 4 in libc-2.9.so[2b83283b9000+168000]
>> [19463.265457] ld[3673]: segfault at 0 ip 00002ad2f7a3a11b sp
>> 00007fffd11023a8 error 4 in libc-2.9.so[2ad2f79b6000+168000]
>> [19474.653491] ld[3680]: segfault at 0 ip 00002b7f10f0e11b sp
>> 00007fffbac8a978 error 4 in libc-2.9.so[2b7f10e8a000+168000]
>> [19507.935271] ld[3687]: segfault at 0 ip 00002af5c9eb511b sp
>> 00007fff12fe00d8 error 4 in libc-2.9.so[2af5c9e31000+168000]
>> [19528.740436] ld[3701]: segfault at 0 ip 00002b265616a11b sp
>> 00007fff2ceb8d98 error 4 in libc-2.9.so[2b26560e6000+168000]
>> [19606.865585] ld[3754]: segfault at 0 ip 00002ae3a079811b sp
>> 00007fff6bc4d3c8 error 4 in libc-2.9.so[2ae3a0714000+168000]
>> [263529.064795] convert[24941]: segfault at 7fffd973b6b8 ip
>> 00007fffdb1e2ed9 sp 00007fffd973b660 error 7 in
>> libMagickCore.so.1.0.0[7fffdb0fe000+1b5000]
>> [268495.776398] convert[28595]: segfault at 7fffc7e2b608 ip
>> 00007fffc9e13ed9 sp 00007fffc7e2b5b0 error 7 in
>> libMagickCore.so.1.0.0[7fffc9d2f000+1b5000]
>>
>>
>>
>> Any advice?
>> Thanks,
>> -Thomas
>>
>> _______________________________________________
>> BLUG mailing list
>> BLUG@linuxfan.com
>> http://mailman.cs.indiana.edu/mailman/listinfo/blug
>
> _______________________________________________
> BLUG mailing list
> BLUG@linuxfan.com
> http://mailman.cs.indiana.edu/mailman/listinfo/blug
>

--
http://resc.smugmug.com/

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] Fwd: possible causes of segfaults

Did you upgrade the system without rebooting it? (That should be
reproducible, though.) Are you using a version of ImageMagick compiled
for a different distribution of Linux? (I know a lot of RPM-based
systems do not bundle a lot of programs.)

If you downloaded an RPM that wasn't compiled specifically for your
distribution/version, I would expect that to be the cause of the
problem. If that's the cause of the problem, building it from source
should clear it up.

If you're using the version of ImageMagick that comes with your
distribution and you've not recently performed an upgrade, memory
corruption seems the most likely candidate. Does that machine have
unparitied memory? Any type of memory other than unparitied would likely
show an error instead of just producing bogus data. (It is why I hate
unparitied memory.)

It could also be a CPU fault.

Problems with hard drives tend to show up as errors with the specific
media. (It'll list the device producing the error.) Hardware problems
regarding media do not normally produce segfaults, unless the
application fails to handle the error case.

My recommendation: Pick an upcoming weekend and tell them the services
of this machine will be unavailable. Then start the memory test at
5:15pm (adjusted for the end of your workday) and run a memory checker
all weekend. Then come in 15 minutes early on Monday to check for errors
and reboot the system.

Cheers,
Steven Black

On Mon, Dec 21, 2009 at 03:58:40PM -0500, Thomas Smith wrote:
> Hiya,
>
> I just installed a new file / computation server at my work, and a few
> days into its tenure, I've started noticing some segfaults.
>
> For instance, I was running a bunch of image processing jobs, and
> ImageMagick's "convert" program segfaulted.  I ran it again on the
> same data, and it did fine.  So I'm wondering if I have bad hardware,
> or bad libraries, or what?
>
> Possible causes of segmentation faults that I know of:
> Hardware --- could be very random and difficult to find, might need to
> totally shut down the server and run a memory tester for days in order
> to find.
>
> Filesystem corruption --- should be reproducible, right?  if "convert"
> segfaults once, it should do it again...
>
> Libraries / os problems --- should be reproducible too, right?
>
>
>
> from dmesg:
> [17217.872070] ld[16027]: segfault at 0 ip 00002ae3b445411b sp
> 00007fffc39a74e8 error 4 in libc-2.9.so[2ae3b43d0000+168000]
> [17354.753195] ld[20115]: segfault at 0 ip 00002b832843d11b sp
> 00007fff729f2fa8 error 4 in libc-2.9.so[2b83283b9000+168000]
> [19463.265457] ld[3673]: segfault at 0 ip 00002ad2f7a3a11b sp
> 00007fffd11023a8 error 4 in libc-2.9.so[2ad2f79b6000+168000]
> [19474.653491] ld[3680]: segfault at 0 ip 00002b7f10f0e11b sp
> 00007fffbac8a978 error 4 in libc-2.9.so[2b7f10e8a000+168000]
> [19507.935271] ld[3687]: segfault at 0 ip 00002af5c9eb511b sp
> 00007fff12fe00d8 error 4 in libc-2.9.so[2af5c9e31000+168000]
> [19528.740436] ld[3701]: segfault at 0 ip 00002b265616a11b sp
> 00007fff2ceb8d98 error 4 in libc-2.9.so[2b26560e6000+168000]
> [19606.865585] ld[3754]: segfault at 0 ip 00002ae3a079811b sp
> 00007fff6bc4d3c8 error 4 in libc-2.9.so[2ae3a0714000+168000]
> [263529.064795] convert[24941]: segfault at 7fffd973b6b8 ip
> 00007fffdb1e2ed9 sp 00007fffd973b660 error 7 in
> libMagickCore.so.1.0.0[7fffdb0fe000+1b5000]
> [268495.776398] convert[28595]: segfault at 7fffc7e2b608 ip
> 00007fffc9e13ed9 sp 00007fffc7e2b5b0 error 7 in
> libMagickCore.so.1.0.0[7fffc9d2f000+1b5000]
>
>
>
> Any advice?
> Thanks,
> -Thomas
>
> _______________________________________________
> BLUG mailing list
> BLUG@linuxfan.com
> http://mailman.cs.indiana.edu/mailman/listinfo/blug

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

[BLUG] Fwd: possible causes of segfaults

Hiya,

I just installed a new file / computation server at my work, and a few
days into its tenure, I've started noticing some segfaults.

For instance, I was running a bunch of image processing jobs, and
ImageMagick's "convert" program segfaulted.  I ran it again on the
same data, and it did fine.  So I'm wondering if I have bad hardware,
or bad libraries, or what?

Possible causes of segmentation faults that I know of:
Hardware --- could be very random and difficult to find, might need to
totally shut down the server and run a memory tester for days in order
to find.

Filesystem corruption --- should be reproducible, right?  if "convert"
segfaults once, it should do it again...

Libraries / os problems --- should be reproducible too, right?

from dmesg:
[17217.872070] ld[16027]: segfault at 0 ip 00002ae3b445411b sp
00007fffc39a74e8 error 4 in libc-2.9.so[2ae3b43d0000+168000]
[17354.753195] ld[20115]: segfault at 0 ip 00002b832843d11b sp
00007fff729f2fa8 error 4 in libc-2.9.so[2b83283b9000+168000]
[19463.265457] ld[3673]: segfault at 0 ip 00002ad2f7a3a11b sp
00007fffd11023a8 error 4 in libc-2.9.so[2ad2f79b6000+168000]
[19474.653491] ld[3680]: segfault at 0 ip 00002b7f10f0e11b sp
00007fffbac8a978 error 4 in libc-2.9.so[2b7f10e8a000+168000]
[19507.935271] ld[3687]: segfault at 0 ip 00002af5c9eb511b sp
00007fff12fe00d8 error 4 in libc-2.9.so[2af5c9e31000+168000]
[19528.740436] ld[3701]: segfault at 0 ip 00002b265616a11b sp
00007fff2ceb8d98 error 4 in libc-2.9.so[2b26560e6000+168000]
[19606.865585] ld[3754]: segfault at 0 ip 00002ae3a079811b sp
00007fff6bc4d3c8 error 4 in libc-2.9.so[2ae3a0714000+168000]
[263529.064795] convert[24941]: segfault at 7fffd973b6b8 ip
00007fffdb1e2ed9 sp 00007fffd973b660 error 7 in
libMagickCore.so.1.0.0[7fffdb0fe000+1b5000]
[268495.776398] convert[28595]: segfault at 7fffc7e2b608 ip
00007fffc9e13ed9 sp 00007fffc7e2b5b0 error 7 in
libMagickCore.so.1.0.0[7fffc9d2f000+1b5000]

Any advice?
Thanks,
-Thomas

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] power off CD?

On Mon, Dec 21, 2009 at 08:19:44AM -0500, David Ernst wrote:
> On Sun, Dec 20, 2009 at 11:27:21PM -0500, Steven Black wrote:
> >On Fri, Dec 18, 2009 at 01:11:29AM -0500, David Ernst wrote:
> >> So, we put the CD into the CD drive of my Ubuntu (Jaunty)
> >> machine... it spins up, and ... my computer turns off. power off. As
> >> if our power had gone out, but it hadn't.
> >
> >You should check the kernel logs. (/var/log/dmesg.*)
>
> I thought the dmesg logs wrote only boot-up info..? I took a look and
> didn't see anything, any clues on what you think I should be looking
> for?

After boot up the kernel writes logs to a pipe and either syslog-ng
or klogd will read from the pipe and write to a file. It should cover
issues that crop up after boot.

This log gets rotated at boot, so an 'ls -al dmesg*' will show you the
dates and times of the last write to each log. That can be used to find
the correct file, then look at the last line or so.

I don't know about your distribution, but my distro time-stamps each line
with time-since-boot information (in seconds?). This allows you to look
for the jump from boot time to post-boot messages.

However, if it was the CMOS/BIOS causing the shutdown (as I have seen
before) it is unlikely there will be any meaningful entries in the log.

> >Powering off is unusual. Now, kernel halting due to some hardware issue,
> >that's a lot more common.
>
> Agreed. But would the computer really turn off its power on a kernel
> panic?

Not normally, no.

I believe there is logic in place to prevent kernel dead-locks which
may reboot the system -- I thought it performed a reboot and didn't
shut it down, though.

Such an incident would be flagged in the kernel log.

> >I've seen low-level errors power off machines before. If you have logs
> >available in your CMOS/BIOS settings that may shine some light on it.
>
> next time I reboot, I'll try to take a look. :) I don't remember
> ever seeing such a thing, though.

These logs are fairly common for server hardware, but rare for
desktop/laptop hardware. Look for anything with "Log" in the name. Just
a forewarning, sometimes these are not directly human-readiable and just
contain time-stamps and hex numbers.

Cheers,
Steven Black

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] power off CD?

On Sun, Dec 20, 2009 at 11:27:21PM -0500, Steven Black wrote:
>On Fri, Dec 18, 2009 at 01:11:29AM -0500, David Ernst wrote:
>> So, we put the CD into the CD drive of my Ubuntu (Jaunty)
>> machine... it spins up, and ... my computer turns off. power off. As
>> if our power had gone out, but it hadn't.
>
>You should check the kernel logs. (/var/log/dmesg.*)

I thought the dmesg logs wrote only boot-up info..? I took a look and
didn't see anything, any clues on what you think I should be looking
for?

>Powering off is unusual. Now, kernel halting due to some hardware issue,
>that's a lot more common.

Agreed. But would the computer really turn off its power on a kernel
panic?

>I've seen low-level errors power off machines before. If you have logs
>available in your CMOS/BIOS settings that may shine some light on it.

next time I reboot, I'll try to take a look. :) I don't remember
ever seeing such a thing, though.

David
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] power off CD?

On Fri, Dec 18, 2009 at 01:11:29AM -0500, David Ernst wrote:
> So, we put the CD into the CD drive of my Ubuntu (Jaunty)
> machine... it spins up, and ... my computer turns off. power off. As
> if our power had gone out, but it hadn't.

You should check the kernel logs. (/var/log/dmesg.*)

Powering off is unusual. Now, kernel halting due to some hardware issue,
that's a lot more common.

I've seen low-level errors power off machines before. If you have logs
available in your CMOS/BIOS settings that may shine some light on it.

Cheers,
Steven Black

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] power off CD?

On Fri, Dec 18, 2009 at 11:50:46AM -0600, Beartooth wrote:
>On Fri, 18 Dec 2009, David Ernst wrote:
>
>> So, my wife Priscilla makes a copy of a data CD. Some friends
>> are in the psychology dept at IU, and she volunteered for an
>> experiment and ended up with some kind of imaging info of her
>> brain. I don't think that's important to the story, but maybe.
>> Anyway, she drags a couple of extra files onto the disk in
>> Windows (Vista), and it's ready to give to her friend. She
>> said to me "can you put this in you computer and make sure that
>> it works?" Sure. [....]
>
> Would the psych department be doing scans using medical
>imaging?

Well, I'm a total outsider here, so I can't speak to exactly what they
are doing. But, they do these experiments where they put you in an
MRI machine (or something) and then ask you to do stuff (listen to
music, answer some questions, etc) and watch what happens in your
brain when you're thinking about it. In addition to the $50 or
whatever, they also give you a picture of your brain. :)

> I have CDs/DVDs of some scans made at local hospitals --
>MRIs and I disremember what else -- which I haven't managed to
>display with Fedora. They don't harm any machine I've tried them
>on, just fail to display imagery.

Yeah, I really don't think it's the images themselves... But I don't
know that.

David
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug