|
Home > Archive > Unix administration > November 2007 > Interviews (was: Re: Documenting a server conf)
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Interviews (was: Re: Documenting a server conf)
|
|
| Dave Hinz 2007-10-19, 7:20 pm |
| On Fri, 19 Oct 2007 06:55:11 -0700, Doug Freyburger <dfreybur@yahoo.com> wrote:
> Dave Hinz <DaveH...@gmail.com> wrote:
[vbcol=seagreen]
> Huh. You've managed to overhear one of my interviews a few
> years ago? Scenario questions are the most rewarding to both
> ask and answer.
I like scenerios that I've lived through, or are mostly what I've lived
through with a bit of artistic license thrown in to make it a better
story or riddle. How about this one:
On your first day as sysadmin for a small department, you are told that
the previous admin quit so abruptly that he left no root password or
support documentation, left his sweater hanging in the computer room and
(...you find later, the hard way) that he left his lunch in the desk
drawer. It's been weeks and now things are starting to break. The
group of users are scientists, not IT folks, so their understanding is
limited to "the box doesn't go bing anymore when I push the button".
Upon entering the computer room, you see perhaps a dozen racks, systems
open and running on benches, under tables, and on a cluttered desk (see
previous re: lunch in drawer). The desk has the nicest monitor in the
room and a jumble of reference books, tapes, and general unixish
clutter.
Where do you start? How do you figure out what is important and what
isn't, how do you get into the domain without a password, and what do
you do first?
(I use this scenerio or a variation of same at nearly every interview,
once the "can the person do the work" has been established and we've
moved on to the "What is their thought process and judgement, and would
I want to spend most of my waking hours with them" phase of the conversation.)
Dave
| |
| Sylvain Robitaille 2007-10-20, 1:31 am |
| Dave Hinz wrote:
> On your first day as sysadmin for a small department, you are told that
> the previous admin quit so abruptly that he left no root password or
> support documentation, ...
> ...
> It's been weeks and now things are starting to break. ...
> ...
> Upon entering the computer room, you see perhaps a dozen racks, systems
> open and running on benches, under tables, and on a cluttered desk (see
> previous re: lunch in drawer). The desk has the nicest monitor in the
> room and a jumble of reference books, tapes, and general unixish
> clutter.
Um ... if I have been there for weeks, why have I not been into the
computer room sooner to investigate what's where and which machines are
running what?
> Where do you start?
What's broken already?
Possibly examining a computer (a colleague's or one of the users')
that I can get access to, to see certain configuration items (what
are the DNS servers, for example, and from there I likely can find a
lot more information; what is the system's default network gateway,
and what information might I be able to glean from it?)
What monitoring stations are there? What are they showing?
> How do you figure out what is important and what isn't, ...
Definitely the DNS is a good place to get some idea of what systems are
on the network, and to some extent which ones are likely to carry at
least some importance. In other words, even if not formally classified
as "documentation", start investigating for some documentation in some
other form.
If stuff is breaking, what symptoms are the users seeing? Can we trace
packets to get a sense of what should be expected?
> how do you get into the domain without a password, ...
"the domain"??? Are you sure you're hiring a unix admin?
> and what do you do first?
What, according to the users and the employer, are the most important
things that are broken?
--
----------------------------------------------------------------------
Sylvain Robitaille syl@alcor.concordia.ca
Systems and Network analyst Concordia University
Instructional & Information Technology Montreal, Quebec, Canada
----------------------------------------------------------------------
| |
| Dave Hinz 2007-10-20, 1:22 pm |
| On Sat, 20 Oct 2007 03:59:38 +0000 (UTC), Sylvain Robitaille <syl@alcor.concordia.ca> wrote:
> Dave Hinz wrote:
>
>
> Um ... if I have been there for weeks, why have I not been into the
> computer room sooner to investigate what's where and which machines are
> running what?
Ah, sorry. They were without an admin for weeks because they figured
they could handle it themselves. So there's been a bit of that sort of
thing going on as well.
[vbcol=seagreen]
> What's broken already?
(toolname) doesn't work.
> Possibly examining a computer (a colleague's or one of the users')
> that I can get access to, to see certain configuration items (what
> are the DNS servers, for example, and from there I likely can find a
> lot more information; what is the system's default network gateway,
> and what information might I be able to glean from it?)
NIS domainname maybe, see what systems it mounts, read the startup
scripts for (toolname), yup.
> What monitoring stations are there? What are they showing?
Not present.
[vbcol=seagreen]
> Definitely the DNS is a good place to get some idea of what systems are
> on the network, and to some extent which ones are likely to carry at
> least some importance. In other words, even if not formally classified
> as "documentation", start investigating for some documentation in some
> other form.
Makes sense.
> If stuff is breaking, what symptoms are the users seeing? Can we trace
> packets to get a sense of what should be expected?
Packet tracing is probably way more complicateded than this would take.
>
> "the domain"??? Are you sure you're hiring a unix admin?
Last I checked NIS still calls it "domainname"... so, about that, what
box do you think would be the best box to break into, and what would
your approach be?
[vbcol=seagreen]
> What, according to the users and the employer, are the most important
> things that are broken?
OK, once the bleeding is stopped, then what?
| |
| Sylvain Robitaille 2007-10-20, 1:22 pm |
| Dave Hinz wrote:
>
> OK, once the bleeding is stopped, then what?
Well, there isn't a single answer to that question. A lot depends on
what the state of the operation is. At the very least I would start
prioritizing among actually documenting the network and its systems and
services (which I would consider very important), adding monitoring
stations (and host-based scripts) so that I could have a chance to
identify new problems before they affect users, resolving issues that
the users see as problems, making sure users have access to all services
they need, and they know how to use those services, etc. (probably in
that order of importance in my mind, but that's something you would need
to run past at least someone above, and likely colleagues as well ...)
--
----------------------------------------------------------------------
Sylvain Robitaille syl@alcor.concordia.ca
Systems and Network analyst Concordia University
Instructional & Information Technology Montreal, Quebec, Canada
----------------------------------------------------------------------
| |
| andrea 2007-10-20, 7:21 pm |
| I hope I understand correctly, but wouldn't be correct to search for
serious bugs in the server "shell-less" and try to get a root shell?
That would really solve everything...
| |
| Dave Hinz 2007-10-21, 1:26 pm |
| On Sat, 20 Oct 2007 14:02:19 -0700, andrea <kerny404@gmail.com> wrote:
> I hope I understand correctly, but wouldn't be correct to search for
> serious bugs in the server "shell-less" and try to get a root shell?
> That would really solve everything...
Yup. So, the OS is completely hosed and you may not have a network. What
are your options? What would you try first?
| |
| Doug Freyburger 2007-10-21, 7:22 pm |
| Dave Hinz <DaveH...@gmail.com> wrote:
>
> On your first day as sysadmin for a small department, you are told that
> the previous admin quit so abruptly that he left no root password or
....
> Where do you start?
The generic job of the sysadmin, cartoon version - Dive face first
into an ocean of chaos and come out soonest on the opposite
shoreline with bundles of organization in your mouth. Lap after
lap until the chaos is reduced to the point your job is boring.
Then train up a junior admin and transfer to someplace melting ...
The point debugging issue can be some of the most fun work. I
love to turn a debugging session by me into a course for the
benefit of the rest of the team on how to mesh the layers of
whatever modules. Start with finding out what hosts are the most
important and what apps on them. Start with a quick physical
audit, hardware check, network review of stuff moving from ping
to netmasks to whatever control stuff they use (NIS on the
production floor is asking for doomsday but fix it today then
reengineer it in a couple of weeks). Then move on layer to
layer.
Have someone send out for Scezchuan food, pizza, twinkies,
jolt cola, dr pepper and coffee. None go in the DC but they do
go into the people who go into the DC. Same triage principle
in action - Switching to broccoli is next week's effort not for the
meltdown time ...
When interviewing Oracle DBAs I like to play the client manager:
My system is too small. How do you want to proceed? The
answers I've gotten have ranged from bull sessions on migration
strategy to SQL application debugging. It's dratted hard to keep
ahead of a good DBA with any consulting experience when
building a scenario in your head to match their questions. For
an SA, not so hard as Dave is showing but still quite exhilarating.
One deal of interviewing - Give good enough interviews that the
folks you say no to ask for a development plan so they can
progress to the point they can work for you. The best in field
advertizing you can make for your firm - They are so good I am
working towards being on their staff because I want to work with
those guys,
| |
| Dave Hinz 2007-10-21, 7:22 pm |
| On Sun, 21 Oct 2007 14:33:33 -0700, Doug Freyburger <dfreybur@yahoo.com> wrote:
> The generic job of the sysadmin, cartoon version - Dive face first
> into an ocean of chaos and come out soonest on the opposite
> shoreline with bundles of organization in your mouth. Lap after
> lap until the chaos is reduced to the point your job is boring.
> Then train up a junior admin and transfer to someplace melting ...
You've seen my resume, apparently?
> The point debugging issue can be some of the most fun work.
OH, hell yes. During the n'th issue last week, I turned to my manager
and said "I enjoy the hell out of this, you know that, right?" He knew.
> I
> love to turn a debugging session by me into a course for the
> benefit of the rest of the team on how to mesh the layers of
> whatever modules. Start with finding out what hosts are the most
> important and what apps on them. Start with a quick physical
> audit, hardware check, network review of stuff moving from ping
> to netmasks to whatever control stuff they use (NIS on the
> production floor is asking for doomsday but fix it today then
> reengineer it in a couple of weeks). Then move on layer to
> layer.
A logical approach for many things, yup.
> Have someone send out for Scezchuan food, pizza, twinkies,
> jolt cola, dr pepper and coffee. None go in the DC but they do
> go into the people who go into the DC. Same triage principle
> in action - Switching to broccoli is next week's effort not for the
> meltdown time ...
Yup. In a disaster recovery or all-hands-on-deck exercise, keep 'em
well fed.
> When interviewing Oracle DBAs I like to play the client manager:
> My system is too small. How do you want to proceed? The
> answers I've gotten have ranged from bull sessions on migration
> strategy to SQL application debugging. It's dratted hard to keep
> ahead of a good DBA with any consulting experience when
> building a scenario in your head to match their questions.
It's a great way to see how people think and communicate. And keeping
the scenerio consistent with itself is challenging, yes.
> For
> an SA, not so hard as Dave is showing but still quite exhilarating.
> One deal of interviewing - Give good enough interviews that the
> folks you say no to ask for a development plan so they can
> progress to the point they can work for you.
I did just that early in my attempts to get into the sysadmin field.
The person who I had interviewed with gave me a dozen or so things to
learn about so that next time we sat across an interview table, I'd be
ready.
A year or so later, a sysadmin job opened up, and I called him to say
"I've been working on learning what you suggested, may I use you as a
reference in this other posting?" He kind of chuckled, sure, no
problem. The interview, it turns out, was with him. This time I had
the answers to the point where I could talk about, say, automounter and
when it's appropriate to use, that sort of thing. Got the job that
time.
> The best in field
> advertizing you can make for your firm - They are so good I am
> working towards being on their staff because I want to work with
> those guys,
We've got a really good guy doing tech screening for us, so by the time
someone comes in for an actual interview, it's more about people skills
and, quite frankly, "here is why our environment is interesting and you
would like it here". We've got some really cool stuff going on and I
don't know of any environments nearly as interesting in the area.
| |
|
| Begin <1193002413.964859.83280@q5g2000prf.googlegroups.com>
On Sun, 21 Oct 2007 14:33:33 -0700, Doug Freyburger <dfreybur@yahoo.com> wrote:
>
> Have someone send out for Scezchuan food, pizza, twinkies,
> jolt cola, dr pepper and coffee. None go in the DC but they do
> go into the people who go into the DC. Same triage principle
> in action - Switching to broccoli is next week's effort not for the
> meltdown time ...
But I LIKE broccoli! In other words, time management is pretty
important. At some point eight hours of sleep and a fresh start
can mean more progress than just trucking on.
Admittedly I've been on the other side, to burnout and beyond. Getting
back is costly and extremely time consuming and in my case was not paid
for by the people that caused it. So I forced myself to care for my own
health first, second, and last. If we're in an actual shooting war it
might be different. Then again, I'm not in the military.
I'm also less inclined to keep people from impaling themselves on their
own wilful stupidity. I won't let it happen if I can reasonably help it,
but at some point I will ha^H^Hregretfully let'em eat twinkies.
[...]
> One deal of interviewing - Give good enough interviews that the
> folks you say no to ask for a development plan so they can
> progress to the point they can work for you. The best in field
> advertizing you can make for your firm - They are so good I am
> working towards being on their staff because I want to work with
> those guys,
:-)
--
j p d (at) d s b (dot) t u d e l f t (dot) n l .
This message was originally posted on Usenet in plain text.
Any other representation, additions, or changes do not have my
consent and may be a violation of international copyright law.
| |
| andrea 2007-11-13, 7:32 am |
| I thought I could do something like that, every time I edit a conf
file (in /etc) before saving I launch a script which backups it in
another folder (with timestamps).
And then I can simply use diff to see what I actually changed.
The thing is, how could I automate that script?
I think I'll have to do it manually, maybe creating a command in
textmate and running i every time...
| |
| Sylvain Robitaille 2007-11-13, 1:27 pm |
| andrea wrote:
> I thought I could do something like that, every time I edit a conf
> file (in /etc) before saving I launch a script which backups it in
> another folder (with timestamps).
> And then I can simply use diff to see what I actually changed.
>
> The thing is, how could I automate that script?
I use the following (t)csh command alias, called "bu":
foreach file (!*)
\cp -pi ${file} \
`dirname ${file}`/old/`basename ${file}`.`date +%Y%m%d`
end
The only assumption (that perhaps shouldn't be made in general, but is
reasonably safe on my systems) is that "`dirname ${file}`/old" exists as
a directory. If not, the copy simply fails, though and I can create the
"old" directory before re-invoking the "bu" command alias. If a backup
of the file already exists for "today", the command will prompt whether
or not it should be over-written. It's almost always correct not to
over-write (and perhaps then manually create a backup with an additional
extension).
I hope this helps ...
--
----------------------------------------------------------------------
Sylvain Robitaille syl@alcor.concordia.ca
Systems and Network analyst Concordia University
Instructional & Information Technology Montreal, Quebec, Canada
----------------------------------------------------------------------
| |
| Thorbjoern Ravn Andersen 2007-11-13, 1:27 pm |
| andrea <kerny404@gmail.com> writes:
> The thing is, how could I automate that script?
Consider using a versioning control system.
--
Thorbjørn Ravn Andersen
| |
| Dave Hinz 2007-11-14, 1:34 am |
| On Tue, 13 Nov 2007 04:08:42 -0800, andrea <kerny404@gmail.com> wrote:
> I thought I could do something like that, every time I edit a conf
> file (in /etc) before saving I launch a script which backups it in
> another folder (with timestamps).
> And then I can simply use diff to see what I actually changed.
> The thing is, how could I automate that script?
Kind of like crontab -e, visudo, and other tools? Basically they're
just a wrapper to an edit which then sanity-checks the resulting edits
to make sure they're valid. No reason you couldn't write these for
other config files, if they're not already out there.
> I think I'll have to do it manually, maybe creating a command in
> textmate and running i every time...
shell script would be plenty good, not sure what textmate is?
| |
| andrea 2007-11-14, 7:33 am |
| On 14 Nov, 04:05, Dave Hinz <DaveH...@gmail.com> wrote:
> On Tue, 13 Nov 2007 04:08:42 -0800, andrea <kerny...@gmail.com> wrote:
>
> Kind of like crontab -e, visudo, and other tools? Basically they're
> just a wrapper to an edit which then sanity-checks the resulting edits
> to make sure they're valid. No reason you couldn't write these for
> other config files, if they're not already out there.
Interesting, you mean creating a command which is a wrapper for the
actual editor and does the backup?
>
>
> shell script would be plenty good, not sure what textmate is?
It's a powerful editor for Macosx, I can send ssh commands through it
very easily.
| |
| andrea 2007-11-14, 7:33 am |
| On 13 Nov, 20:14, Thorbjoern Ravn Andersen <nospam0...@gmail.com>
wrote:
> andrea <kerny...@gmail.com> writes:
>
> Consider using a versioning control system.
> --
> Thorbj=F8rn Ravn Andersen
I considered but for example svn is a bit too "big" for this purpouse.
I should create a repository on that machine, and then a working
directory and exporting on /etc when modified something.
The thing is that I just need information about diffs from the
original files, not all the informations svn gives me...
| |
|
| Begin <1195042232.162478.98230@o3g2000hsb.googlegroups.com>
On Wed, 14 Nov 2007 04:10:32 -0800, andrea <kerny404@gmail.com> wrote:
> On 13 Nov, 20:14, Thorbjoern Ravn Andersen <nospam0...@gmail.com> wrote:
>
> I considered but for example svn is a bit too "big" for this purpouse.
> I should create a repository on that machine, and then a working
> directory and exporting on /etc when modified something.
>
> The thing is that I just need information about diffs from the
> original files, not all the informations svn gives me...
There are quite the few of those around, both "big" and less so.
Subversion has tremendous momentum as purported CVS replacement,
amounting to hype well beyond its capabilites.
I'd be tempted to try and use whatever came with my system for
simplicity's sake, in casu CVS and RCS (on which CVS builds). It is true
that RCS has a long history. But it also is pretty light and perfect
for cases where all you need is track one file, or multiple independent
files in the same directory, as opposed to directories or entire trees
of directories with related files. Since it deals with only one file at
a time it is relatively easy to move files under its control around:
just make sure the matching ,v file makes the same relative movement.
--
j p d (at) d s b (dot) t u d e l f t (dot) n l .
This message was originally posted on Usenet in plain text.
Any other representation, additions, or changes do not have my
consent and may be a violation of international copyright law.
| |
| Thorbjoern Ravn Andersen 2007-11-14, 1:28 pm |
| andrea <kerny404@gmail.com> writes:
> I considered but for example svn is a bit too "big" for this purpouse.
> I should create a repository on that machine, and then a working
> directory and exporting on /etc when modified something.
You would have a central repository.
> The thing is that I just need information about diffs from the
> original files, not all the informations svn gives me...
In that case you may want to just tar up /etc and give the tarball a
date stamp. Then you can compare whenever you need the data.
--
Thorbjørn Ravn Andersen
| |
| Dave Hinz 2007-11-14, 7:21 pm |
| On Wed, 14 Nov 2007 04:07:29 -0800, andrea <kerny404@gmail.com> wrote:
> On 14 Nov, 04:05, Dave Hinz <DaveH...@gmail.com> wrote:
[vbcol=seagreen]
> Interesting, you mean creating a command which is a wrapper for the
> actual editor and does the backup?
Yup. Someone else mentioned a good datestamping method, combine 'em all
into a script. Make a datestamped copy, open the editor, and if you
want to get fancy, do whatever that tool's "now check my config file for
sanity" before exiting.
>
> It's a powerful editor for Macosx, I can send ssh commands through it
> very easily.
Ah, OK, I just use the native Terminal.app and vi, but everyone has
their own editor of choice.
| |
| Dave Hinz 2007-11-14, 7:21 pm |
| On 14 Nov 2007 17:32:57 +0100, Thorbjoern Ravn Andersen <nospam0000@gmail.com> wrote:
> andrea <kerny404@gmail.com> writes:
>
>
> You would have a central repository.
We're using Sun Explorer for historical info on config files in our
(1600+ node) environment, works quite well. Once a month we run an
explorer, it dumps config and runtime info from each system. Very
useful to see how a box looked a month or year ago. All the explorers
go to a common, nfs mounted filesystem off on a NetApp somewhere.
|
|
|
|
|