Sunday, September 27, 2009

Re: [BLUG] storage options

On Sunday 27 September 2009 3:37:10 am Joe Auty wrote:
> Hey Guys,
>
>
> I'm looking for some perspectives on the storage options available
> today, and on the horizon...
>
>
> Specifically, I'm thinking about something that would be a little
> handier and more future proof for my servers I run. Specifically, I run
> VMWare Server on Linux and several virtual machines all on the same
> hardware.
>
> There are several different variables and technologies at this point:
>
> - direct attach w. software or hardware RAID
>
> - file systems such as ZFS and BTRFS and the cool stuff you can do with
> them
>
> - NAS
>
> - external RAID arrays
>
> - iSCSI
>
> - getting higher quality disks such as SCSI and solid state drives
>
>
>
> In a perfect world, here is what I'd like:
>
>
> - A storage device not tethered to my servers so that I can add more
> storage as needed without planning for downtime and running dd, and
> upgrade server hardware without having to reinvest in storage.
>
> - Fast I/O suitable for running VMs. There are tricks that can be
> employed which will help one get by with SATA, but you still have to be
> careful not to do heavy disk operations on the host as to render the
> guests unresponsive. This is a nuisance, and of course makes
> hypothetically backing up complete VM disk images difficult.
>
> - Disk redundancy, i.e. some sort of RAID configuration.
>
> - Some sort of solution somewhere between Cletus the Slackjaw Yokel's
> Windows XP box and high level enterprise stuff. I like the idea of
> spreading out the I/O demand across several cheap disks, adding more
> disk as needed, replacing failed disks as necessary. I don't need crazy
> fast I/O, I just need something a little more than a single SATA disk.
> So, enterprise level technology is not what I'm after here, just
> convenience and decent performance.
>
> - Something that will be relatively future proof and not cost me a
> fortune :)
>
> - I'm using nearly 250 gig right now so my capacity needs are not
> tremendous, but it would cost me a lot more than what I'm paying now to
> run within a VPS provider such as Linode or Slicehost, and I'd rather
> not get into the position where my costs grow significantly just to add
> a modest amount of disk space. My storage needs don't warrant an
> expensive SAN such as the ones that are no doubt in use by these
> providers, one of my VMs is running Windows, and I like the control I
> have now and I like working with my server provider. So, I'm not really
> keen on moving my eight servers to VMs provided by one of these companies
>
> - A way to backup all of this data (snapshots), preferably via the same
> overall design so that I have something I can test with and perhaps even
> fail over to in the event of an emergency. I know that offsite
> redundancy is the golden egg for many companies, but hey, this is a
> perfect world type wishlist! :) I use Amazon S3 for an offsite backup in
> addition to my current backup to backup my most important data, but to
> keep my costs to a minimum I'd prefer to just stay with rsync to
> multiple cheap disks.
>
>
> I'm interested in learning more about iSCSI, and am fascinated with
> BTRFS and ZFS. Do any of you have any experience with any of this, have
> any general recommendations, thoughts, predictions, anything? I don't
> need to buy anything tomorrow, I'm just thinking that it would be nice
> to think ahead a little.
>
> It seems like I'm kind of stuck in between not needing to invest heavily
> in storage like a big company would, yet I'm pushing the limits of
> run-of-the-mill consumer grade direct attach SATA type stuff - the kind
> of solution that would be great for using with Time Machine to store
> pictures of your kids.
>

You might try looking into using freenas. it can do alot of the things you
are looking for and has support for zfs and iscsi at this point.
--
PGP e-mail is welcome! Get my 1024 bit signature key from:
<http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x00D1EABB>

[BLUG] storage options

Hey Guys,


I'm looking for some perspectives on the storage options available
today, and on the horizon...


Specifically, I'm thinking about something that would be a little
handier and more future proof for my servers I run. Specifically, I run
VMWare Server on Linux and several virtual machines all on the same
hardware.

There are several different variables and technologies at this point:

- direct attach w. software or hardware RAID

- file systems such as ZFS and BTRFS and the cool stuff you can do with
them

- NAS

- external RAID arrays

- iSCSI

- getting higher quality disks such as SCSI and solid state drives

In a perfect world, here is what I'd like:


- A storage device not tethered to my servers so that I can add more
storage as needed without planning for downtime and running dd, and
upgrade server hardware without having to reinvest in storage.

- Fast I/O suitable for running VMs. There are tricks that can be
employed which will help one get by with SATA, but you still have to be
careful not to do heavy disk operations on the host as to render the
guests unresponsive. This is a nuisance, and of course makes
hypothetically backing up complete VM disk images difficult.

- Disk redundancy, i.e. some sort of RAID configuration.

- Some sort of solution somewhere between Cletus the Slackjaw Yokel's
Windows XP box and high level enterprise stuff. I like the idea of
spreading out the I/O demand across several cheap disks, adding more
disk as needed, replacing failed disks as necessary. I don't need crazy
fast I/O, I just need something a little more than a single SATA disk.
So, enterprise level technology is not what I'm after here, just
convenience and decent performance.

- Something that will be relatively future proof and not cost me a
fortune :)

- I'm using nearly 250 gig right now so my capacity needs are not
tremendous, but it would cost me a lot more than what I'm paying now to
run within a VPS provider such as Linode or Slicehost, and I'd rather
not get into the position where my costs grow significantly just to add
a modest amount of disk space. My storage needs don't warrant an
expensive SAN such as the ones that are no doubt in use by these
providers, one of my VMs is running Windows, and I like the control I
have now and I like working with my server provider. So, I'm not really
keen on moving my eight servers to VMs provided by one of these companies

- A way to backup all of this data (snapshots), preferably via the same
overall design so that I have something I can test with and perhaps even
fail over to in the event of an emergency. I know that offsite
redundancy is the golden egg for many companies, but hey, this is a
perfect world type wishlist! :) I use Amazon S3 for an offsite backup in
addition to my current backup to backup my most important data, but to
keep my costs to a minimum I'd prefer to just stay with rsync to
multiple cheap disks.


I'm interested in learning more about iSCSI, and am fascinated with
BTRFS and ZFS. Do any of you have any experience with any of this, have
any general recommendations, thoughts, predictions, anything? I don't
need to buy anything tomorrow, I'm just thinking that it would be nice
to think ahead a little.

It seems like I'm kind of stuck in between not needing to invest heavily
in storage like a big company would, yet I'm pushing the limits of
run-of-the-mill consumer grade direct attach SATA type stuff - the kind
of solution that would be great for using with Time Machine to store
pictures of your kids.


--
Joe Auty
NetMusician: web publishing software for musicians
http://www.netmusician.org
joe@netmusician.org
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Sunday, September 13, 2009

[BLUG] 3.5"/5.25" drive brackets

Off topic, but people on here are likely to have some. I used to have
like 20 pairs of 3.5" to 5.25" drive brackets and they would just get in
the way, now I can't find any and I need some. Does anyone have a pair
of them or preferably 3 pairs? Please e-mail me directly. Thanks.

Mark

--
Mark Krenz
Bloomington Linux Users Group
http://www.bloomingtonlinux.org/
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Thursday, September 10, 2009

Re: [BLUG] IPtables (don't forget IPv6!)

If you're using iptables by hand, you need to either (1) block all IPv6
traffic, or (2) perform similar blocks on the IPv6 side.

I'm a big fan of using other products to make it more painless to manage
firewalls. (There's a potential replacement to iptables *and* iptables6,
and if/when things change I don't want to deal with it.)

I've been just disabling IPv6, but that was purely a short-term solution
until the firewall product I use supported IPv6.

My product of choice is currently Shorewall. It has the advantage of
also running on my Linksys wireless router. Current versions support
both IPv4 and IPv6.

Cheers,
Steven Black


On Thu, Sep 10, 2009 at 02:56:08PM +0000, Mark Krenz wrote:
>
> I've never done any tests to see what is most efficient, but if you
> have a large number of drops for a specific port, it might be useful to
> send a single port to its own chain, then deal with the drops for more
> specific hosts and networks in that chain. So if you had a seperate
> chain for port 80, packets destined for port 25 or whatever would skip
> over checking all the port 80 rules.
>
> So something like this:
>
>
> /sbin/iptables -N port80
> /sbin/iptables -A port80 -s 64.1.2.3/32 -d 0/0 -j DROP
> /sbin/iptables -A port80 -s 24.5.6.0/24 -d 0/0 -j DROP
> /sbin/iptables -A port80 -s 10.0.0.0/8 -d 0/0 -j DROP
> /sbin/iptables -A port80 -s 0/0 -d 0/0 -j ACCEPT
>
> /sbin/iptables -A INPUT -p tcp -s 0/0 -d 0/0 --dport 80 -j port80
> /sbin/iptables -A INPUT -p tcp -s 0/0 -d 0/0 --dport 25 -j ACCEPT
> /sbin/iptables -A INPUT -p tcp -s 0/0 -d 0/0 -j DROP
>
>
>
> So I think now that port 25 packets will only have to check 2 rules
> instead of 5 because they won't have to go through the chain of port 80
> rules. Now I'm curious to see how much of a difference it makes. I'll
> have to test it.
>
>
> On Thu, Sep 10, 2009 at 02:23:10PM GMT, Scott Blaydes [sblaydes@gmail.com] said the following:
> > Okay, I am starting to get into some really long DROP lists in my
> > IPtables config and was wondering what others on the list thought
> > about best placement of the DROPs vs ACCEPTs. Do you have your drops
> > early in the file so that they blocks IPs get denied as quickly as
> > possible or do you have your allows earlier in the file?
> >
> > Is one way really more efficient than the other?
> >
> > Thank you,
> > Scott Blaydes

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] IPtables

I've never done any tests to see what is most efficient, but if you
have a large number of drops for a specific port, it might be useful to
send a single port to its own chain, then deal with the drops for more
specific hosts and networks in that chain. So if you had a seperate
chain for port 80, packets destined for port 25 or whatever would skip
over checking all the port 80 rules.

So something like this:


/sbin/iptables -N port80
/sbin/iptables -A port80 -s 64.1.2.3/32 -d 0/0 -j DROP
/sbin/iptables -A port80 -s 24.5.6.0/24 -d 0/0 -j DROP
/sbin/iptables -A port80 -s 10.0.0.0/8 -d 0/0 -j DROP
/sbin/iptables -A port80 -s 0/0 -d 0/0 -j ACCEPT

/sbin/iptables -A INPUT -p tcp -s 0/0 -d 0/0 --dport 80 -j port80
/sbin/iptables -A INPUT -p tcp -s 0/0 -d 0/0 --dport 25 -j ACCEPT
/sbin/iptables -A INPUT -p tcp -s 0/0 -d 0/0 -j DROP

So I think now that port 25 packets will only have to check 2 rules
instead of 5 because they won't have to go through the chain of port 80
rules. Now I'm curious to see how much of a difference it makes. I'll
have to test it.


On Thu, Sep 10, 2009 at 02:23:10PM GMT, Scott Blaydes [sblaydes@gmail.com] said the following:
> Okay, I am starting to get into some really long DROP lists in my
> IPtables config and was wondering what others on the list thought
> about best placement of the DROPs vs ACCEPTs. Do you have your drops
> early in the file so that they blocks IPs get denied as quickly as
> possible or do you have your allows earlier in the file?
>
> Is one way really more efficient than the other?
>
> Thank you,
> Scott Blaydes
> _______________________________________________
> BLUG mailing list
> BLUG@linuxfan.com
> http://mailman.cs.indiana.edu/mailman/listinfo/blug
>

--
Mark Krenz
Bloomington Linux Users Group
http://www.bloomingtonlinux.org/
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

[BLUG] IPtables

Okay, I am starting to get into some really long DROP lists in my
IPtables config and was wondering what others on the list thought
about best placement of the DROPs vs ACCEPTs. Do you have your drops
early in the file so that they blocks IPs get denied as quickly as
possible or do you have your allows earlier in the file?

Is one way really more efficient than the other?

Thank you,
Scott Blaydes
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Wednesday, September 9, 2009

Re: [BLUG] tracking downloads of a single file

On Tue, Sep 08, 2009 at 10:46:04AM -0400, Ben Shewmaker wrote:
> Second, what stats program(s) do others on the list use?* I like Google
> Analytics, but I'm curious if there are other notable programs that run
> locally vs. a tracking service in the cloud.* Any good open source ones?*
> I've been using Analog and Google so far. . .

Just an FYI...

I run my system with cookies disabled for Google Analytics. It's like
opting out of the Google Analytics program. Don't trust the numbers.

I'm sure I'm not the only one that runs things like that. Plus, you're
talking about downloads of specific files. `wget` and the like will
ignore cookies.

Cheers,
Steven Black

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug