Friday, July 20, 2007

[BLUG] Bloomingpedia presentation

I know this isn't Linux, but I wanted to remind everyone that we're
having a presentation on Bloomingpedia tommorow at the Monroe County
Public Library Auditorium starting at 2pm. You are all welcome to
attend. Chris Eller and I will be presenting. More information is
available here:

http://www.bloomingpedia.org/wiki/Bloomingpedia:Introductory_Presentation_2007

Thanks,
Mark

--
Mark Krenz
Bloomington Linux Users Group
http://www.bloomingtonlinux.org/
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Wednesday, July 18, 2007

[BLUG] Quicktime VR stuff

I know this is quite off topic and doesn't involve Linux at all, but
would anybody here happen to know of anybody local that does
panoramic/Quicktime VR type stuff? I have a client who is interested in
including some movies on her site, and I'm not really connected with the
local photographer scene...


--
Joe Auty
NetMusician: web publishing software for musicians
http://www.netmusician.org
joe@netmusician.org
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Monday, July 16, 2007

Re: [BLUG] Running out of memory during log turnover

Is this always happening on the same time of the day?

If so, check your crontab(s) to verify that there isn't an unrelated reason
for memory to be low that time of night.

I know you're seeing the problem with Apache, but have you checked your other
log files? Sometimes a relatively minor warning in another logfile, at the
time that you're having the issue with main problem, can help you pinpoint
what is going on.

Cheers,
Steven Black

On Sat, Jul 14, 2007 at 11:29:19PM -0400, Joe Auty wrote:
> Hello,
>
> I'm wondering if anybody knows what I might do to address this problem.
> I'm using FreeBSD and Apache is having a hard time when my Apache error
> log files turn over causing Apache to crash with the following output to
> my error log... No other logs of any sort on the system cause this
> problem, but this happens consistently with my Apache error log:

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] Unix conventions for controlling file access

As others have said, this technique is decades old. There is nothing harmful
about it.

This technique is older than Linux, let alone the Ext2 file-system. Every
file-system for Linux has to support this, as it is an inherent
characteristic of Unix-like file-systems and our ability to do upgrades of
live systems (and perform rm -r / and actually delete everything).

Most security measures are about reducing risk. If you only have a temp. file
as an open-able file for a short period of time, it reduces the risk of
others snooping it on you. -- Especially when the names are picked truly
randomly.

Cheers,
Steven Black

On Fri, Jul 13, 2007 at 04:05:49PM +0000, Mark Krenz wrote:
> # lsof | grep deleted
> [snip]
> mysqld 31958 mysql 7u REG 9,0 0 15 /tmp/ibwZfdKe (deleted)
> mysqld 31958 mysql 13u REG 9,0 0 16 /tmp/ibR0K4tQ (deleted)
> mysqld 31959 mysql 6u REG 9,0 0 14 /tmp/ibuPNjWc (deleted)
> mysqld 31959 mysql 7u REG 9,0 0 15 /tmp/ibwZfdKe (deleted)
> mysqld 31959 mysql 13u REG 9,0 0 16 /tmp/ibR0K4tQ (deleted)
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Saturday, July 14, 2007

[BLUG] Running out of memory during log turnover

Hello,

I'm wondering if anybody knows what I might do to address this problem.
I'm using FreeBSD and Apache is having a hard time when my Apache error
log files turn over causing Apache to crash with the following output to
my error log... No other logs of any sort on the system cause this
problem, but this happens consistently with my Apache error log:

Jul 14 23:00:00 netmusician newsyslog[81819]: logfile turned over due to
size>5000K
Out of memory during request for 32 bytes, total sbrk() is 10196992 bytes!
Callback called exit at
/usr/local/lib/perl5/site_perl/5.8.8/HTML/Element.pm line 1502.
BEGIN failed--compilation aborted at
/usr/local/lib/perl5/site_perl/5.8.8/HTML/Formatter.pm line 106.
Callback called exit at
/usr/local/lib/perl5/site_perl/5.8.8/HTML/Formatter.pm line 106.
BEGIN failed--compilation aborted at
/usr/local/lib/perl5/site_perl/5.8.8/HTML/FormatText.pm line 68.
Callback called exit at
/usr/local/lib/perl5/site_perl/5.8.8/HTML/FormatText.pm line 68.
BEGIN failed--compilation aborted at
/usr/local/rt3/lib/RT/Transaction_Overlay.pm line 84.
Callback called exit at /usr/local/rt3/lib/RT/Transaction_Overlay.pm
line 84.
BEGIN failed--compilation aborted at
/usr/local/rt3/lib/RT/Transactions.pm line 73.
Callback called exit at /usr/local/rt3/lib/RT/Transactions.pm line 73.
BEGIN failed--compilation aborted at /usr/local/rt3/lib/RT/Record.pm
line 1490.
Callback called exit at /usr/local/rt3/lib/RT/Record.pm line 1490.
BEGIN failed--compilation aborted at
/usr/local/rt3/lib/RT/CurrentUser.pm line 72.
Callback called exit at /usr/local/rt3/lib/RT/CurrentUser.pm line 72.
BEGIN failed--compilation aborted at /usr/local/rt3/lib/RT.pm line 49.
Callback called exit at /usr/local/rt3/lib/RT.pm line 49.
BEGIN failed--compilation aborted at /usr/local/rt3/bin/webmux.pl line 66.
Callback called exit at /usr/local/rt3/bin/webmux.pl line 66.

It looks like RT is a part of this problem consistently each time this
has happened... RT is setup to use mod_perl.

I've tried various Apache tuning techniques thinking that there was some
sort of memory leak, but I haven't been able to isolate this problem.
Since it is a PITA to manually have to restart Apache every time this
happens, I'm hoping that some of you may have some ideas as to some
things I can try here?

The other reason why I think this might be an Apache problem is because
of the several constant error messages that I see in my error log such as:

[Sat Jul 14 23:11:12 2007] [info] Subsequent (No.2) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:12 2007] [info] Subsequent (No.3) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:12 2007] [info] Subsequent (No.4) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:12 2007] [info] Subsequent (No.5) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:12 2007] [info] Subsequent (No.6) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:13 2007] [info] Subsequent (No.7) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:13 2007] [info] Subsequent (No.8) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.9) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.10) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.11) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Connection to child 3 established
(server netmusician.org:443, client 74.130.30.11)
[Sat Jul 14 23:11:14 2007] [info] Seeding PRNG with 136 bytes of entropy
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.12) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Initial (No.1) HTTPS request received
for child 3 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.13) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.2) HTTPS request
received for child 3 (server netmusician.org:443)
[Sat Jul 14 23:11:14 2007] [info] Subsequent (No.14) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:15 2007] [info] Subsequent (No.3) HTTPS request
received for child 3 (server netmusician.org:443)
[Sat Jul 14 23:11:15 2007] [info] Subsequent (No.15) HTTPS request
received for child 2 (server netmusician.org:443)
[Sat Jul 14 23:11:15 2007] [info] Subsequent (No.4) HTTPS request
received for child 3 (server netmusician.org:443)
[Sat Jul 14 23:11:30 2007] [info] (70007)The timeout specified has
expired: SSL input filter read failed.
[Sat Jul 14 23:11:30 2007] [info] Connection to child 2 closed with
standard shutdown(server netmusician.org:443, client 74.130.30.11)
[Sat Jul 14 23:11:30 2007] [info] (70007)The timeout specified has
expired: SSL input filter read failed.
[Sat Jul 14 23:11:30 2007] [info] Connection to child 3 closed with
standard shutdown(server netmusician.org:443, client 74.130.30.11)

I'm not entirely convinced that my Apache install is healthy, although
it seems to behave just fine and I've noticed no real problems with
Apache. It has been perfectly stable, outside of this reproducible crash
caused by log file turnover.

I'm kind of grasping at straws here, I know.


--
Joe Auty
NetMusician: web publishing software for musicians
http://www.netmusician.org
joe@netmusician.org
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Friday, July 13, 2007

Re: [BLUG] Unix conventions for controlling file access

On Fri, Jul 13, 2007 at 04:16:50PM GMT, Dave Monnier REN-ISAC [dmonnier@ren-isac.net] said the following:
>
> So I wonder, how does this come into play with reserve blocks, quotas, etc?
> I wonder how much stuff depends on intact metadata.

Fortunately, it doesn't seem to be affected by it. At least in Linux
2.2.19 (machine I have access to that has quotas turned on):

Here is a program I quickly wrote in perl to do this:

-------------------------------------------------------------------------------
#!/usr/bin/perl -w

# This is a test to see if I can defeat quota limits by deleting a file
# with an open file handle.

open(FILE, ">/home/markk/largefile");

$i = 0;
$limit = 4600000;
while ($i < $limit) {
print FILE "0";
$i++;
}

sleep 10;
unlink('/home/markk/largefile');

sleep 300;
-------------------------------------------------------------------------------

$ ./test.pl &
[1] 2012
$ ls -l largefile
-rw-r--r-- 1 markk business 393216 Jul 13 13:42 largefile
$ ls -lh largefile
-rw-r--r-- 1 markk business 4.2M Jul 13 13:43 largefile
$ ls -lh largefile
-rw-r--r-- 1 markk business 4.4M Jul 13 13:43 largefile
$ quota -v
Disk quotas for user markk (uid 4505):
Filesystem blocks quota limit grace files quota limit
/dev/sdb1 4640 5000 5500 27 0 0
$ ls -lh largefile
ls: largefile: No such file or directory
$ quota -v
Disk quotas for user markk (uid 4505):
Filesystem blocks quota limit grace files quota limit
/dev/sdb1 4640 5000 5500 27 0 0
$ dd if=/dev/zero of=anotherfile bs=1M count=1000
/home: write failed, user disk quota exceeded too long.
/home: write
failed, user disk quota exceeded too long.
dd:
anotherfile: Disk quota exceeded
1+0 records in
0+0 records out

$ jobs
[1]+ Running ./test.pl &
$ kill %
[1]+ Terminated ./test.pl
$
$ quota -v
Disk quotas for user markk (uid 4505):
Filesystem blocks quota limit grace files quota limit
/dev/sdb1 472 5000 5500 27 0 0

I'm guessing that this also doesn't work in 2.6, but probably checking
is not a bad idea. It would be interesting if it did work and you could
somehow write a program that used FUSE to have a hidden filesystem
inside a deleted file that exceeded your quota.


--
Mark Krenz
Bloomington Linux Users Group
http://www.bloomingtonlinux.org/
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] Unix conventions for controlling file access

On Fri, 2007-07-13 at 12:16 -0400, Dave Monnier REN-ISAC wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Mark Krenz wrote:
>
> > Its interesting to note that mysql seems to do this too:
>
> Wow, sure enough.
>
> So I wonder, how does this come into play with reserve blocks, quotas, etc?
> I wonder how much stuff depends on intact metadata.
>

Quotas, block counts, etc are all handled at the inode level -- not the
directory level. So they're just normal files. They just don't have a
name in the filesystem.

You can play around with it by doing something like this in perl:
-------------------------------------------
#!/usr/bin/perl

open(H,">foo");
print H "hello world!\n";
dumpstat("open filehandle",H);
link("foo","foo.link");
dumpstat("two links to filehandle",H);
unlink("foo");
dumpstat("foo is now gone",H);
unlink("foo.link");
dumpstat("foo.link is now also gone",H);
# the file still exists and has data in it
close(H);
# the OS now cleans up, since there are no links and
# the only process with it open has closed it.

sub dumpstat {
my($message,$thing)=@_;
my($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,
$atime,$mtime,$ctime,$blksize,$blocks)
= stat(*$thing);
print $message,"\n";
print " inode: $ino, nlink: $nlink\n";
}
------------------------------

The filesystem uses the inode as the basis for all file operations. The
names are just syntactic sugar for us mere mortals! There's a couple of
side effects of this design decision:

* Each file (the contents & metadata) is uniquely identifiable by a
pair: (device number, inode number). On the above perl script (and
using the 'stat' program) we can see this information:

File: `inode_fun'
Size: 549 Blocks: 16 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 4718621 Links: 1
Access: (0664/-rw-rw-r--) Uid: (11907/bdwheele) Gid: ( 500/bdwheele)
Access: 2007-07-13 12:34:07.000000000 -0400
Modify: 2007-07-13 12:34:03.000000000 -0400
Change: 2007-07-13 12:34:03.000000000 -0400

The only thing that's in the directory entry is the name 'inode_fun' and
the inode number. The 'Device' part comes from the OS, and everything
else comes from the inode.

* any number of names can point to a single set of data. Unlike DOS
complaining about "cross-linked files" when running chkdsk, having
multiple names for a file is a feature.

* mv only changes the name by adding a new link in the destination and
removing the old link in the source. No data is copied. This is why on
early versions of unix you couldn't mv across filesystem boundaries (the
inode numbers are not unique across devices). GNU's mv handles this
case by making a copy.

* The modify and change times, which seem redundant, actually measure
different things. The change time (ctime) is when the inode (meta) data
was changed: ownership, permissions, size, etc. Modification time
(mtime) is when the content was changed. Usually changing the content
changes the file time, so mtime and ctime are the same, but that's not
necessarily true.

* that goofy remove a file while its open thing works :)

* if you don't create enough inodes at filesystem creation time, you
could end up where you couldn't create a new file even though there were
disk blocks free. Its not a big deal these days, but it happened to me
on floppy disks a couple of times, since you want to conserve as much
space as possible by not "wasting" space for the inode tables. df -i
will give you the inode statistics.


That's all I can think of off the top of my head. Hopefully it put
everyone into a dreamy Friday afternoon sleep!

Brian

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug