|
Home > Archive > Red Hat General > October 2005 > A way to determine if a file is open?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
A way to determine if a file is open?
|
|
| Grant Schoep 2005-10-06, 6:03 pm |
| Not sure where to start or post this question. So just taking a few random
wacks at this.
Is there a systematic way to determine if a file is currently "open" by a
live process somewhere? I don't really care where or by what. Just that
some process actually has that file open.
Problem. We have this logging directory that a number of different apps on
this system share. They are all bad processes that like to leave these
logging files around. The data is important, I want to archive it off at
certain points. But I don't want to "move" away those files in the
directory if they are still open. There is no way by looking at the data in
the file itself to know when it is closed.
I just want to be able to look at a filename and somehow determine if some
process has it open. Note, this could be on another machine. As this is an
NFS mount point.
It doesn't matter if the solution is via C, some scripting languange, shell
language, or whatever. I just am trying to find out if it is even possible
to determine if a file is open or not.
Since it probably matters, in this case all our machines that would have
one of these files open are running RedHat ES 3. All are NFS mounting that
partition which is using the ext3 filesystem.
| |
| Simon Bone 2005-10-07, 7:49 am |
| On Thu, 06 Oct 2005 22:42:58 +0000, Grant Schoep wrote:
> Not sure where to start or post this question. So just taking a few random
> wacks at this.
>
> Is there a systematic way to determine if a file is currently "open" by a
> live process somewhere? I don't really care where or by what. Just that
> some process actually has that file open.
>
Try lsof.
> Problem. We have this logging directory that a number of different apps on
> this system share. They are all bad processes that like to leave these
> logging files around. The data is important, I want to archive it off at
> certain points. But I don't want to "move" away those files in the
> directory if they are still open. There is no way by looking at the data in
> the file itself to know when it is closed.
>
Really the apps need to follow some sort of locking protocol. You would
follow the same protocol if you knew it. Do you have the source code?
One common protocol is to put a file into /var/lock/<something> with the
PID of the process currently using the file. If that lock exists no other
process should use the log, but if not then a process that wants the lock
can try to create it. Of course there are problems here- it just shifts
the issue from excluding simultaneous access to a big file (e.g.a log) to
a small one. This protocol has particular problems with local caching over
NFS mounts. But just what (if anything) is done in your case is something
you need to look into.
>
> Since it probably matters, in this case all our machines that would have
> one of these files open are running RedHat ES 3. All are NFS mounting
> that partition which is using the ext3 filesystem.
If you can't work this out from here, try contacting Redhat for technical
support. They can probably give you a useful answer for this specific
case. You do need to ensure that each of those machines is not
simultaneously going to write to one NFS mounted file, or else chaos will
ensue. Your software will hopefully be doing that already, and you should
really only need to check you do the same as it is.
HTH,
Simon Bone
| |
| Grant Schoep 2005-10-24, 3:44 pm |
| Thanks. Its all good now. I used a combo of the output from lsof, and
direct Ruby, manip of directory entries. Ruby rocks, this was my first Ruby
tool. Its another scripting language like Perl, but is more directly object
oriented, and, well, its pretty readable and reusable too. We now can
accuratly type "ls *.log" without it saying argument list too long. This
rules.I actually wrote two scripts,one in c-shell, and one in Ruby. On our
one system with +30,000 plus files, it took Ruby about 10 seconds to sort
those files out in a year_month directory structure. I restored it, and
with csh, it took almost 7 minutes.
| |
| Nate Smith 2005-10-24, 3:44 pm |
| Grant Schoep wrote:
> Not sure where to start or post this question. So just taking a few random
> wacks at this.
might be better posed in the c language group.
>
> Is there a systematic way to determine if a file is currently "open" by a
> live process somewhere? I don't really care where or by what. Just that
> some process actually has that file open.
you could write a process that tries a low-level open EXCLUSIVELY.
then loop until successfully attaching it exclusively. at that point
copy it off into your scratch viewing area somewhere and then release
the exclusive grip on it. note that if you are running in real time
and need to release the file quickly, you may have to do your action
in a timely way or offload the action via copying to a temp file.
or just have some tail -f window running in an xterm window on it.
scroll through that window (making sure to give it zillions of scroll
lines first). depends on how you want to use the logs. are you able
to cut down the volume with a filter like grep? then pass that into
the tail -f.
>
> Problem. We have this logging directory that a number of different apps on
> this system share. They are all bad processes that like to leave these
> logging files around. The data is important, I want to archive it off at
> certain points. But I don't want to "move" away those files in the
> directory if they are still open. There is no way by looking at the data in
> the file itself to know when it is closed.
>
> I just want to be able to look at a filename and somehow determine if some
> process has it open. Note, this could be on another machine. As this is an
> NFS mount point.
yikes.
>
> It doesn't matter if the solution is via C, some scripting languange, shell
> language, or whatever. I just am trying to find out if it is even possible
> to determine if a file is open or not.
>
> Since it probably matters, in this case all our machines that would have
> one of these files open are running RedHat ES 3. All are NFS mounting that
> partition which is using the ext3 filesystem.
ok - this makes it harder. trying to open exclusively may only work
on a file local to the current machine, as the operating system has
no idea what other machines may be doing at the moment. the tail -f
window will work for viewing, as new log lines will be appended and you
will see them arrive. but now if you want to use data in the lines to
trigger other processes, you probably want to pipe it into a watchdog
app that can do the triggering.
but you want to archive off chunks of each of these files....
wait - i seem to remember now that the exclusive lock will work.
a c program can do it. you loop until you get the exclusive open.
then you append the file to your archive file. if you call
system("cat foo.log >> foolog.archive") you may have trouble when
the child process system makes runs into the exclusive grip.
so you could do a read/write loop on the whole file to append it,
then open the file a second time with TRUNCate to zap out all the
archived stuff from the active file, close it, and release it to
the hounds.
so why not just have a "tail -f foo.log >> foolog.archive" window
running in the back ground all day? because you never zap the
original back to zero length and the disks get full....
i dont know how bad things get if the original file would be
instead a softlink to the archive non-NFSed file off on the archive
machine and one of the other machines NFSed to that link went
bellyup on you. probably nothing good.
- nate
|
|
|
|
|