Unix Programming - How do you make a prog wait for a file to be ready before reading?

Author

How do you make a prog wait for a file to be ready before reading?

DyslexicAnaboko

2006-12-07, 7:36 pm

Hello forum,

I have a program that towards its end creates and writes to a file
called "S1."

I then need my program to read the newly created file and print it out
to stdout.

I have already written both parts, the file is indeed created and
written to and I know the code I wrote for reading the file and
printing it to stdout works because I tested it.

My problem is that the program tries to read the file before it even
exists and I don't know how to make it not do this.

The reason I know this is my problem is because I put the reading
process to sleep for 10 seconds ( sleep(10) ; ) and as a result the
program executed properly. It actually printed the file to stdout.

However I cannot assume that it will always take 10 seconds for the
file to be ready, so I was curious as to what any one in the forum
would do?

I would include code, but it is extensive and would just complicate
things because it is long and it does a lot more than I described.

Thank you for your time,

Eli Hayon

Stephane CHAZELAS

2006-12-07, 7:36 pm

2006-12-7, 13:55(-08), DyslexicAnaboko:
> Hello forum,
>
> I have a program that towards its end creates and writes to a file
> called "S1."
>
> I then need my program to read the newly created file and print it out
> to stdout.
>
> I have already written both parts, the file is indeed created and
> written to and I know the code I wrote for reading the file and
> printing it to stdout works because I tested it.
[...]

Why not using a pipe instead of a regular file? Maybe a named
pipe? See mkfifo(2/3), mkfifo(1), mknod(2)

--
Stéphane

Pascal Bourguignon

2006-12-07, 7:36 pm

"DyslexicAnaboko" <DyslexicAnaboko@gmail.com> writes:
> I have a program that towards its end creates and writes to a file
> called "S1."
>
> I then need my program to read the newly created file and print it out
> to stdout.
>
> I have already written both parts, the file is indeed created and
> written to and I know the code I wrote for reading the file and
> printing it to stdout works because I tested it.
>
> My problem is that the program tries to read the file before it even
> exists and I don't know how to make it not do this.

You'd have to use some IPC (Inter Process Communication). Stephane
suggested a pipe. Other possibilities exist. One possibility, which
is easy to implement and to use in a wide range of situations is tu
use a token file.

The consumer must wait until it finds a token file exists. When it
finds the token file, it can delete it and process the data file.

The producer writes the data file, and when it completes, it creates
the token file.

file="/tmp/data"
token="${file}.token"

# consummer:
while [ ! -r "$token" ] ; do sleep 10 ; done ; rm "$token" ; process "$file"

# producer:
produce > "$file" && touch "$token"

A variation of this scheme is to use the data file itself as token
file, but the producer must create it under a different name, and
rename(2) it to the token name.

file="/tmp/data"
temp="${file}.temp"

# consummer:
while [ ! -r "$file" ] ; do sleep 10 ; done ; process "$file" && rm "$file"

# producer:
produce > "$temp" && mv "$temp" "$file"

Note: it's important that the $temp and $file path be on the same
device, for the mv to be atomic!

--
__Pascal Bourguignon__ http://www.informatimago.com/

WARNING: This product attracts every other piece of matter in the
universe, including the products of other manufacturers, with a
force proportional to the product of the masses and inversely
proportional to the distance between them.

Barry Margolin

2006-12-08, 1:33 am

In article <1165528540.486154.184820@80g2000cwy.googlegroups.com>,
"DyslexicAnaboko" <DyslexicAnaboko@gmail.com> wrote:

> Hello forum,
>
> I have a program that towards its end creates and writes to a file
> called "S1."
>
> I then need my program to read the newly created file and print it out
> to stdout.
>
> I have already written both parts, the file is indeed created and
> written to and I know the code I wrote for reading the file and
> printing it to stdout works because I tested it.
>
> My problem is that the program tries to read the file before it even
> exists and I don't know how to make it not do this.

It should try to open the file in a loop, to wait for the open() to
return successfully. E.g.

while ((fd = open(filename, O_RDONLY)) == -1 && errno == ENOENT)
sleep(1);

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

DyslexicAnaboko

2006-12-08, 1:33 am

I appreciate everyone's replies and I have tried out some of the above
mentioned methods. There are some things I cannot alter because it
would force me to deviate from a specification set by my professor. I
need to create the file S1 and then I have to open it and print it
out... it has to be done like that unfortunately otherwise I would have
just used "| tee S1" and let the data output to stdout.

I have discovered yet another problem... my program doesn't only try to
read from a file that doesn't exist, it tries to read from a file that
is being written to, which in turn gives me no output.

Are there any recommendations for the following idea:

How do I tell my program to wait for a file that is currently being
written to, to wait until the file is idle before trying to open it?

So in other words, process1 is writing to a file called S1 and finishes
in X seconds, process2 should only try to access/open/read S1 after it
has stopped being written to (idle).

Any ideas are appreciated.

I am sorry if I am asking questions that may seem silly to ask, I am a
novice at C and Unix (I am used to java), this is for a class that I am
taking, however I am learning a lot.

Thank you for your time,

Eli Hayon

matevzb

2006-12-08, 7:38 am

DyslexicAnaboko wrote:
<snip>
> Are there any recommendations for the following idea:
>
> How do I tell my program to wait for a file that is currently being
> written to, to wait until the file is idle before trying to open it?
>
> So in other words, process1 is writing to a file called S1 and finishes
> in X seconds, process2 should only try to access/open/read S1 after it
> has stopped being written to (idle).
>
> Any ideas are appreciated.
A not-so-pretty solution would be to have S1 send a signal (kill()) to
S2 when it stops writing, for example SIGUSR1. S2 would block/sleep
until the signal is received and once received, read the file. It's not
the best solution or a very good one, but it should work (and it's
probably the easiest to implement).

--
WYCIWYG - what you C is what you get

Pascal Bourguignon

2006-12-08, 7:38 am

"DyslexicAnaboko" <DyslexicAnaboko@gmail.com> writes:

> I appreciate everyone's replies and I have tried out some of the above
> mentioned methods. There are some things I cannot alter because it
> would force me to deviate from a specification set by my professor. I
> need to create the file S1 and then I have to open it and print it
> out... it has to be done like that unfortunately otherwise I would have
> just used "| tee S1" and let the data output to stdout.
>
> I have discovered yet another problem... my program doesn't only try to
> read from a file that doesn't exist,

You must be doing something really wrong. When you try to open a file
for reading, if it doesn't exist, you should get an error. There's
absolutely no way you can "try" to read from a file that doesn't
exist. Read: man 2 open

{
try_again:
int fd=open("file",O_RDONLY);
if(fd<0){
switch(errno){
case ENOENT:
/* the file doesn't exist yet */
sleep(1);
goto try_again;
default:
report_the_other_error(errno);
}
}else{
read_the_file(fd);
}
}

> it tries to read from a file that
> is being written to, which in turn gives me no output.

If you cannot use a token file or any other IPC, you can write an
end-of-file mark in the file. Until you read it, you know that the
file is not finished writting. But your consumer must be prepared to
read the file while it's being written, that is, once it reaches the
end of the file, if it hasn't read the end-of-file mark, then it must
try to read again:

/* pseudo-code */
read_the_file(int fd){
char record[record_size];
bool eof=false;
while(!eof){
int res=read(fd,record,sizeof(record));
if(res<0){
process_the_error(errno);
}else if(0==res){
/* we've reached the end of the file, but not read
the end-of-file mark; let's wait and try again */
sleep(1);
}else if(is_end_of_file_mark(record)){
eof=true;
}else{
process_the_record(record);
}
}
}

More work is needed to handle correctly the record size (if
res<record_size), depending on the kind of file and "records" it may
be a text file with variable length lines separated by a newline, etc.

If you have records, you can have a special record type for the
end-of-file mark. If you have lines, your end-of-file mark must be a
line that cannot occur as normal data. It could be just "END", if
your protocol ensures that any data line that consists only of "END"
be substituted by "\END" for example, and the reader does the inverse
substitution. (And of course any data line containing only "\\\...\\END"
be susbtituted by "\\\...\\\END".)

> Are there any recommendations for the following idea:
>
> How do I tell my program to wait for a file that is currently being
> written to, to wait until the file is idle before trying to open it?
>
> So in other words, process1 is writing to a file called S1 and finishes
> in X seconds, process2 should only try to access/open/read S1 after it
> has stopped being written to (idle).

Never use delays to synchronize processes, it doesn't work.

> Any ideas are appreciated.
>
> I am sorry if I am asking questions that may seem silly to ask, I am a
> novice at C and Unix (I am used to java), this is for a class that I am
> taking, however I am learning a lot.

Thankfully! ;-)

--
__Pascal Bourguignon__ http://www.informatimago.com/

WARNING: This product warps space and time in its vicinity.

Bill Marcum

2006-12-08, 1:23 pm

On 7 Dec 2006 13:55:40 -0800, DyslexicAnaboko
<DyslexicAnaboko@gmail.com> wrote:
> Hello forum,
>
> I have a program that towards its end creates and writes to a file
> called "S1."
>
> I then need my program to read the newly created file and print it out
> to stdout.
>
> I have already written both parts, the file is indeed created and
> written to and I know the code I wrote for reading the file and
> printing it to stdout works because I tested it.
>
> My problem is that the program tries to read the file before it even
> exists and I don't know how to make it not do this.
>
> The reason I know this is my problem is because I put the reading
> process to sleep for 10 seconds ( sleep(10) ; ) and as a result the
> program executed properly. It actually printed the file to stdout.
>
> However I cannot assume that it will always take 10 seconds for the
> file to be ready, so I was curious as to what any one in the forum
> would do?
>
> I would include code, but it is extensive and would just complicate
> things because it is long and it does a lot more than I described.
>
One simple way is for the writing program to rename the file when it is
done writing, then the reader can loop until the renamed file exists.

--
Gordon's first law:
If a research project is not worth doing, it is not worth doing well.

DyslexicAnaboko

2006-12-09, 1:37 am

Once again, thank you all for your help. Everyone's input was helpful,
I finally got it working. This project sure has been mighty difficult,
but it is finally complete...

Thank you all for your time and concern,

Eli Hayon

Brian C

2006-12-15, 1:34 am

Pascal Bourguignon wrote:
> "DyslexicAnaboko" <DyslexicAnaboko@gmail.com> writes:

> try_again:
> int fd=open("file",O_RDONLY);
> if(fd<0){
> switch(errno){
> case ENOENT:
> /* the file doesn't exist yet */
> sleep(1);
> goto try_again;
> default:
> report_the_other_error(errno);
> }
> }else{
> read_the_file(fd);
> }
> }
I do agree with the method. However, I would really suggest a while loop
over a goto. Only in rare cases would I use a goto, and I don't think I
have in ages.

For example:

while(1) // this program I guess gets cancelled by Ctrl+C
{
int fd=open("file",O_RDONLY);
if(fd<0)
{
switch(errno)
{
case ENOENT:
{
sleep(1);
continue;
}
default: report_the_other_error(errno);
}
}
else
read_the_file(fd);
}

Jim Cochrane

2006-12-16, 1:20 pm

On 2006-12-15, Brian C <brianc@no.sp.a.m.dynamissoftware.com> wrote:
> Pascal Bourguignon wrote:
>
> I do agree with the method. However, I would really suggest a while loop
> over a goto. Only in rare cases would I use a goto, and I don't think I
> have in ages.
>
> For example:
>
> while(1) // this program I guess gets cancelled by Ctrl+C
> {
> int fd=open("file",O_RDONLY);
> if(fd<0)
> {
> switch(errno)
> {
> case ENOENT:
> {
> sleep(1);
> continue;
> }
> default: report_the_other_error(errno);
> }
> }
> else
> read_the_file(fd);
> }

Looks like you're missing either a break statement or a while
termination condition, assuming the intent is to duplicate the logic of
the goto version.

--