BLUG: 03/19/08

Wednesday, March 19, 2008

Re: [BLUG] Program/Suite for passing jobs between machines?

I checked out the description of mcfeely.

Both cfengine and Puppet are not designed with that orientation in mind.
My understanding of them is that they're organized more in a manner
which presents definitions of how things should be set up, and has the
system update whatever is needed to make it so.

My script only does part of the process. It doesn't keep track of what ran
where, and doesn't retry anything. It is a fire-and-forget approach. (Well,
with logging.) I could see my script being adapted to such an end, though.

Cheers,
Steven Black

On Wed, Mar 19, 2008 at 02:48:52PM +0000, Mark Krenz wrote:
>
> That's a good example of what you start out with. But eventually what
> happens is that other people need to help create/maintain users (freeing
> up the sysadmin a bit). If you want to write something like a web
> control panel, how do you handle passing requests safely between hosts
> and having dependencies, handling failure properly and so on?
>
> At Suso, I'm doing a lot of custom things beyond what something like
> useradd can do and we'll be doing a lot more that prevents us from using
> things like Cpanel, Plesk, etc. So I have to write my own. But what
> nobody seems to tackle is the task of safely passing jobs and
> information between servers, without just writing a hack. This is what
> Mcfeely was aiming at (http://www.kiva.net/~systhug/mcfeely/). I would
> think that systems like this would be as prevent as window managers on
> Freshmeat, but there appear to be none.
>
> Mark
>
> On Wed, Mar 19, 2008 at 01:42:08PM GMT, Simón Ruiz [simon.a.ruiz@gmail.com] said the following:
> > We use a tiny BASH script to create Samba users on our domain
> > controller, which all workstations authenticate off of, and then fire
> > off an ssh "useradd -m $username" to the mail server if it's a new
> > staff person.
> >
> > It's about the simplest, puniest example of what you're talking about,
> > but since more than 99% of people here only log in to workstations,
> > and the rest of us are sysadmins, it works.
> >
> > Simón
> >
> > On Wed, Mar 19, 2008 at 8:49 AM, Mark Krenz <mark@slugbug.org> wrote:
> > > Anyone have any input?
> > >
> > >
> > > --
> > > Mark Krenz
> > > Bloomington Linux Users Group
> > >

http://www.bloomingtonlinux.org/
> > > _______________________________________________
> > > BLUG mailing list
> > > BLUG@linuxfan.com
> > >

http://mailman.cs.indiana.edu/mailman/listinfo/blug
> > >
> >
> > _______________________________________________
> > BLUG mailing list
> > BLUG@linuxfan.com
> > http://mailman.cs.indiana.edu/mailman/listinfo/blug
> >
>
> --
> Mark Krenz
> Bloomington Linux Users Group
> http://www.bloomingtonlinux.org/
>
> _______________________________________________
> BLUG mailing list
> BLUG@linuxfan.com
> http://mailman.cs.indiana.edu/mailman/listinfo/blug
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] Program/Suite for passing jobs between machines?

You know, when I heard about what you want, my first thought was
something like cfengine or puppet.

At this point, when I want to run something across my cluster, to
install users or do something else, I use a pyexpect script I wrote.

It creates a deliverable (typically either archives a directory,
or a single script) SSH's to the machine (possibly SSH'ing through
intermediaries first), then SCP's back to the starting machine to get
the deliverable. It then extractes the package (if it was packaged)
and runs the script, compiling the stdout to a report on the starting
machine.

It is based on principles I used back in my youth running tests across
the machines in a Quality Assurance lab.

This sort of approach is great for running one-off scripts everywhere.
(This made it easy to check how my machines would handle the DST
changes. It also make it easy for me to find which other machines
had Maxtor drives, when I had two of them fail in relatively quick
succession recently.)

My script to create accounts simply checks the hostname and does
different things based upon where it is run.

If you decide on something like Puppet or cfengine, both have a lot of
activity around them. Puppet is based on Ruby, which can be a plus or
minus depending on your feelings for the language. However, it leverages
more standard components than cfengine.

If my script or something derived from my script sounds more
appealing to you, let me know and we can talk offline about it. I've
implimented solutions like this in both (TCL-based) Expect, as well as
(Python-based) Pexpect.

I must note, though, that I personally plan to migrate to something
like cfengine or Puppet in the future. (Currently, I'm leaning toward
cfengine due to a philosophical disagreement with some of the Ruby
folks.)

Cheers,
Steven Black

On Wed, Mar 19, 2008 at 12:49:33PM +0000, Mark Krenz wrote:
>
> I'm curious if anyone out there has something special that they use
> for passing jobs between machines. Say if you want to run a program on
> machine A to create a user and then it submits jobs on machine B, C and
> D to create the user's mailbox, website, etc., which may be on different
> machines.
>
> Does anyone here do anything like that? I'd be very interested to
> know so that I don't re-invent the wheel.
>
> When I worked at Kiva, we had a great system written by Matt Liggett
> called Mcfeely that was a job queuing system for passing jobs over ssh
> tunnels. Then we had a set of perl modules called Kiva::User written by
> Adrian Hosey and Chris Dent that allowed us to do sophisticated creation
> and transform operations on users. It was really quite remarkable, but
> now its a bit antiquated and unmaintained. When it was written (~1999)
> we couldn't find anything that did the same and I've still never seen
> anything like it. I've been needing to do this stuff on Suso for the
> past few years, but Mcfeely no longer compiles under new gcc/glibc
> versions. However, I'm not sure that Mcfeely is really needed. Jobs
> could be passed directly over an ssh session or through a queuing
> system, but its more up to the implementer. Kinda like whether you want
> to run postfix or just SSMTP. They both still implement SMTP, but they
> have different goals.
>
> So I'm thinking of writing my own job submission/communication suite
> that is not so much a set of programs as it would be a set of standards
> for using an XML DTD to submit jobs information and guidelines for
> security such as authenticating the sender, signing requests and
> responses. After a trial phase I'm willing to release this stuff as
> open source. I was thinking of calling it Newman (Mcfeely was named
> after the famous mailman on Three's Company).
>
> Anyone have any input?
>
>
> --
> Mark Krenz
> Bloomington Linux Users Group
> http://www.bloomingtonlinux.org/
> _______________________________________________
> BLUG mailing list
> BLUG@linuxfan.com
> http://mailman.cs.indiana.edu/mailman/listinfo/blug
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] Program/Suite for passing jobs between machines?

On Wed, Mar 19, 2008 at 05:10:21PM GMT, Simón Ruiz [simon.a.ruiz@gmail.com] said the following:
> On Wed, Mar 19, 2008 at 10:48 AM, Mark Krenz <mark@slugbug.org> wrote:
> >
> > That's a good example of what you start out with. But eventually what
> > happens is that other people need to help create/maintain users (freeing
> > up the sysadmin a bit).
>
> *GASP* You promise?!
>
> ;-)

Heh, no I can't promise you that. I feel your pain though.

--
Mark Krenz
Bloomington Linux Users Group
http://www.bloomingtonlinux.org/

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] Program/Suite for passing jobs between machines?

On Wed, Mar 19, 2008 at 03:18:02PM GMT, Joe Auty [joe@netmusician.org] said the following:
>
> It would seem to me that the big challenge here is generalizing enough
> to be useful in many different environments, without becoming too
> complex in the process. There are a few different approaches here:
> SOAP interfaces, XML, a dedicated specialized service, and then on top
> of this many different languages that various environments aim to
> standardize on. It would be very difficult, I think, to write an Apple-
> style "just add water" service along these lines, nor should you, but
> then at this point the question becomes whether there is a reasonable
> ROI in going the extra mile to generalize enough so that the tool has
> the potential to be useful in other environments. Does this make sense?
>

Cool, opposition. That's the first sign that something may work. ;-)
I know your not opposed to it, but its good to get some critique.

I've read some about SOAP, XML-RPC and that stuff before, but its not
what I'm after. Or at least, their descriptions imply a programmer
centric focus. Running programs with root previliges on other machines
requires a bit more attention to security than most programmers care
about. That's fine, I guess we'll just have to make something new. It
will use XML to make a markup language for transmitting information
about users. I think I'm going to have to practice XML a bit first
before I can make a decision about whether this should be a general
remote program running system or something more specific such as what
I'm after, which is to manage user information. The XML would be
different. If it was specific, then the XML DTD for everything would be
huge, which tags for every kind of data. Imagine if this was a system
that could be used for creating users, but could also be used to pass
information to a 3d raytracing program on a remote system as well. I
think its going to have to focus on user management. But even at that,
there are lots of different tags that could be created, including custom
ones that maybe only one location may use. Hmmmm.

--
Mark Krenz
Bloomington Linux Users Group
http://www.bloomingtonlinux.org/
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] Program/Suite for passing jobs between machines?

On Wed, Mar 19, 2008 at 10:48 AM, Mark Krenz <mark@slugbug.org> wrote:
>
> That's a good example of what you start out with. But eventually what
> happens is that other people need to help create/maintain users (freeing
> up the sysadmin a bit).

*GASP* You promise?!

;-)

Simón

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] Program/Suite for passing jobs between machines?

On Wed, Mar 19, 2008 at 02:55:44PM GMT, Beartooth [karhunhammas@Lserv.com] said the following:
> >>
> >> Try novalug (subscriptions at
> >>http://calypso.tux.org/cgi-bin/mailman/listinfo/novalug), svlug,
> >>or maybe tclug -- the latter two are both on Gmane.
>
> Because novalug and svlug (Northern Virginia and Silicon
> Valley) have lots of subscribers who do that kind of thing.
>

Ok, thanks. The problem I have with that though is that there are
also lots of people here in Bloomington that do these kinds of things
too. After all, Mcfeely was made here in Bloomington. Of course, all
the people that worked on it have moved away except for me and Adrian.

But my goal with the post was to get people here involved in this kind
of stuff. I know there are plenty of sysadmins on the list from viewing
the subscription list. Maybe they aren't reading the list anymore. Who
knows. If I don't get much interest here, I may seek a wider audience.

Re: [BLUG] Program/Suite for passing jobs between machines?

On Mar 19, 2008, at 10:48 AM, Mark Krenz wrote:

>
> That's a good example of what you start out with. But eventually
> what
> happens is that other people need to help create/maintain users
> (freeing
> up the sysadmin a bit). If you want to write something like a web
> control panel, how do you handle passing requests safely between hosts
> and having dependencies, handling failure properly and so on?
>
> At Suso, I'm doing a lot of custom things beyond what something like
> useradd can do and we'll be doing a lot more that prevents us from
> using
> things like Cpanel, Plesk, etc. So I have to write my own. But what
> nobody seems to tackle is the task of safely passing jobs and
> information between servers, without just writing a hack. This is what
> Mcfeely was aiming at (http://www.kiva.net/~systhug/mcfeely/). I
> would
> think that systems like this would be as prevent as window managers on
> Freshmeat, but there appear to be none.
>

It would seem to me that the big challenge here is generalizing enough
to be useful in many different environments, without becoming too
complex in the process. There are a few different approaches here:
SOAP interfaces, XML, a dedicated specialized service, and then on top
of this many different languages that various environments aim to
standardize on. It would be very difficult, I think, to write an Apple-
style "just add water" service along these lines, nor should you, but
then at this point the question becomes whether there is a reasonable
ROI in going the extra mile to generalize enough so that the tool has
the potential to be useful in other environments. Does this make sense?

> Mark
>
> On Wed, Mar 19, 2008 at 01:42:08PM GMT, Simón Ruiz [simon.a.ruiz@gmail.com
> ] said the following:
>> We use a tiny BASH script to create Samba users on our domain
>> controller, which all workstations authenticate off of, and then fire
>> off an ssh "useradd -m $username" to the mail server if it's a new
>> staff person.
>>
>> It's about the simplest, puniest example of what you're talking
>> about,
>> but since more than 99% of people here only log in to workstations,
>> and the rest of us are sysadmins, it works.
>>
>> Simón
>>
>> On Wed, Mar 19, 2008 at 8:49 AM, Mark Krenz <mark@slugbug.org> wrote:
>>> Anyone have any input?
>>>
>>>
>>> --
>>> Mark Krenz
>>> Bloomington Linux Users Group
>>> http://www.bloomingtonlinux.org/
>>> _______________________________________________
>>> BLUG mailing list
>>> BLUG@linuxfan.com
>>> http://mailman.cs.indiana.edu/mailman/listinfo/blug
>>>
>>
>> _______________________________________________
>> BLUG mailing list
>> BLUG@linuxfan.com
>> http://mailman.cs.indiana.edu/mailman/listinfo/blug
>>
>
> --
> Mark Krenz
> Bloomington Linux Users Group
> http://www.bloomingtonlinux.org/
>
> _______________________________________________
> BLUG mailing list
> BLUG@linuxfan.com
> http://mailman.cs.indiana.edu/mailman/listinfo/blug

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] Program/Suite for passing jobs between machines?

On Wed, 19 Mar 2008, Mark Krenz wrote:

>
> Huh? Why?
>
> On Wed, Mar 19, 2008 at 01:39:20PM GMT, Beartooth [karhunhammas@Lserv.com] said the following:
>>
>> Try novalug (subscriptions at
>> http://calypso.tux.org/cgi-bin/mailman/listinfo/novalug), svlug,
>> or maybe tclug -- the latter two are both on Gmane.

Because novalug and svlug (Northern Virginia and Silicon
Valley) have lots of subscribers who do that kind of thing.

--
Beartooth Implacable, Curmudgeonly Codger Learning Linux
On the Internet, you can never tell who is a dog --
supposing you care -- but you can tell who has a mind.
_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] Program/Suite for passing jobs between machines?

That's a good example of what you start out with. But eventually what
happens is that other people need to help create/maintain users (freeing
up the sysadmin a bit). If you want to write something like a web
control panel, how do you handle passing requests safely between hosts
and having dependencies, handling failure properly and so on?

At Suso, I'm doing a lot of custom things beyond what something like
useradd can do and we'll be doing a lot more that prevents us from using
things like Cpanel, Plesk, etc. So I have to write my own. But what
nobody seems to tackle is the task of safely passing jobs and
information between servers, without just writing a hack. This is what
Mcfeely was aiming at (http://www.kiva.net/~systhug/mcfeely/). I would
think that systems like this would be as prevent as window managers on
Freshmeat, but there appear to be none.

Mark

On Wed, Mar 19, 2008 at 01:42:08PM GMT, Simón Ruiz [simon.a.ruiz@gmail.com] said the following:
> We use a tiny BASH script to create Samba users on our domain
> controller, which all workstations authenticate off of, and then fire
> off an ssh "useradd -m $username" to the mail server if it's a new
> staff person.
>
> It's about the simplest, puniest example of what you're talking about,
> but since more than 99% of people here only log in to workstations,
> and the rest of us are sysadmins, it works.
>
> Simón
>
> On Wed, Mar 19, 2008 at 8:49 AM, Mark Krenz <mark@slugbug.org> wrote:
> > Anyone have any input?
> >
> >
> > --
> > Mark Krenz
> > Bloomington Linux Users Group
> >

http://www.bloomingtonlinux.org/
> > _______________________________________________
> > BLUG mailing list
> > BLUG@linuxfan.com
> >

http://mailman.cs.indiana.edu/mailman/listinfo/blug
> >
>
> _______________________________________________
> BLUG mailing list
> BLUG@linuxfan.com
> http://mailman.cs.indiana.edu/mailman/listinfo/blug
>

--
Mark Krenz
Bloomington Linux Users Group
http://www.bloomingtonlinux.org/

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] Program/Suite for passing jobs between machines?

Huh? Why?

On Wed, Mar 19, 2008 at 01:39:20PM GMT, Beartooth [karhunhammas@Lserv.com] said the following:
>
> Try novalug (subscriptions at
> http://calypso.tux.org/cgi-bin/mailman/listinfo/novalug), svlug,
> or maybe tclug -- the latter two are both on Gmane.
>
> --
> Beartooth Implacable, Curmudgeonly Codger Learning Linux
> On the Internet, you can never tell who is a dog --
> supposing you care -- but you can tell who has a mind.
> _______________________________________________
> BLUG mailing list
> BLUG@linuxfan.com
> http://mailman.cs.indiana.edu/mailman/listinfo/blug
>

Re: [BLUG] Program/Suite for passing jobs between machines?

We use a tiny BASH script to create Samba users on our domain
controller, which all workstations authenticate off of, and then fire
off an ssh "useradd -m $username" to the mail server if it's a new
staff person.

It's about the simplest, puniest example of what you're talking about,
but since more than 99% of people here only log in to workstations,
and the rest of us are sysadmins, it works.

Simón

On Wed, Mar 19, 2008 at 8:49 AM, Mark Krenz <mark@slugbug.org> wrote:
> Anyone have any input?
>
>
> --
> Mark Krenz
> Bloomington Linux Users Group
>

http://www.bloomingtonlinux.org/
> _______________________________________________
> BLUG mailing list
> BLUG@linuxfan.com
>

http://mailman.cs.indiana.edu/mailman/listinfo/blug
>

_______________________________________________
BLUG mailing list
BLUG@linuxfan.com
http://mailman.cs.indiana.edu/mailman/listinfo/blug

Re: [BLUG] Program/Suite for passing jobs between machines?

On Wed, 19 Mar 2008, Mark Krenz wrote:

>
> I'm curious if anyone out there has something special that
> they use for passing jobs between machines. Say if you want to
> run a program on machine A to create a user and then it submits
> jobs on machine B, C and D to create the user's mailbox,
> website, etc., which may be on different machines.
>
> Does anyone here do anything like that? I'd be very
> interested to know so that I don't re-invent the wheel.
[snip!]

Try novalug (subscriptions at

http://calypso.tux.org/cgi-bin/mailman/listinfo/novalug), svlug,
or maybe tclug -- the latter two are both on Gmane.

[BLUG] Program/Suite for passing jobs between machines?

I'm curious if anyone out there has something special that they use
for passing jobs between machines. Say if you want to run a program on
machine A to create a user and then it submits jobs on machine B, C and
D to create the user's mailbox, website, etc., which may be on different
machines.

Does anyone here do anything like that? I'd be very interested to
know so that I don't re-invent the wheel.

When I worked at Kiva, we had a great system written by Matt Liggett
called Mcfeely that was a job queuing system for passing jobs over ssh
tunnels. Then we had a set of perl modules called Kiva::User written by
Adrian Hosey and Chris Dent that allowed us to do sophisticated creation
and transform operations on users. It was really quite remarkable, but
now its a bit antiquated and unmaintained. When it was written (~1999)
we couldn't find anything that did the same and I've still never seen
anything like it. I've been needing to do this stuff on Suso for the
past few years, but Mcfeely no longer compiles under new gcc/glibc
versions. However, I'm not sure that Mcfeely is really needed. Jobs
could be passed directly over an ssh session or through a queuing
system, but its more up to the implementer. Kinda like whether you want
to run postfix or just SSMTP. They both still implement SMTP, but they
have different goals.

So I'm thinking of writing my own job submission/communication suite
that is not so much a set of programs as it would be a set of standards
for using an XML DTD to submit jobs information and guidelines for
security such as authenticating the sender, signing requests and
responses. After a trial phase I'm willing to release this stuff as
open source. I was thinking of calling it Newman (Mcfeely was named
after the famous mailman on Three's Company).

Anyone have any input?

Wednesday, March 19, 2008

Blog Archive