My program reads 10000 files in turn (fopen(), fread(), ... fclose())

Regardless of how low its priority (nice'ness), it makes a
machine very very unresponsive if it is running in the
background.
What seems to happen is that the OS, linux, caches each
file. After a while, everything else that was cached gets wiped
out. When a user hits a keystroke, the pages from the user's
application have been paged out and have to be found again.

There is no point in caching the 10000 files read by my
program. They are only read once.
Is there any way to ask the system to *not* to cache a file after
fopen()ing it ?
The 10000 files are NFS mounted, but I imagine the problem would
still happen with a local file system.

Pretty much everything in the code is ANSI C, but this is so
desperate, I would gladly put in linux specific, lower level code
if it helped. I do not want to replace the fopen(), but I am
happy to use fileno() to get the file descriptor and play with
it. I certainly have not found anything useful amongst the calls
to fcntl().
This code has to run in the background on machines maintained by
others, so it is not really practical to play with kernel options
which manipulate the split of vm / file buffer cache.

I would be grateful for any advice.
Andrew

[ Post a follow-up to this message ]

Andrew Torda <andrew.torda@anu.edu.au> writes:

> Is there any way to ask the system to *not* to cache a file after
> fopen()ing it ?

Try posix_fadvise(2) with   POSIX_FADV_SEQUENTIAL|POSIX_FADV_NOREUSE



Regards,
Roger

--
Roger Leigh
Printing on GNU/Linux?  http://gimp-print.sourceforge.net/
Debian GNU/Linux        http://www.debian.org/
GPG Public Key: 0x25BFB848.  Please sign and encrypt your mail.

[ Post a follow-up to this message ]

Andrew Torda wrote:
> My program reads 10000 files in turn (fopen(), fread(), ... fclose())
>
> Regardless of how low its priority (nice'ness), it makes a
> machine very very unresponsive if it is running in the
> background.
> What seems to happen is that the OS, linux, caches each
> file. After a while, everything else that was cached gets wiped
> out. When a user hits a keystroke, the pages from the user's
> application have been paged out and have to be found again.

The same happens when you watch a movie.  It's sad that "modern"
OSes don't really assign priorities to files.  Files that are
obviously only read sequentially (no need for that POSIX function)
shouldn't be cached at all, especially if they are big.
Sequential reading from disk is fast, after all.

Files that are read lots of times in a time interval might be kept
in RAM in preference to applications, though.

Well, that's just my opinion.  Nobody pays me to design and
implement a decent OS ;)

[ Post a follow-up to this message ]