I'm trying to test the throughput of a data processing app by making
numersous copies of the same test file (renamed with a one-up number)
and measuring the time it takes to get the files through the system.
I'm also trying to make sure files are being processed in order.

The files  (DATA.0001.xml, DATA.0002.xml ... DATA.1000.xml) are placed
in a directory which the app is polling.  I'm getting unexpected
results with regard to the order the files are processed.  The app is
polling the directory and grabbing all the "stable" files, they are
sorted by the file's mtime (if the mtime is the same for 2 files, the
filename is used).

In checking the directory with 1000 test files, an ls -ltr (sort by
time) produces unexpected results.  I'd expect the files to be listed
in the order they were created (i.e., consistent with the one-up
number), but this is not
the case.  I did some
digging and it seems the "time last modified" on a file is not
initialized to the creation time.  So, I added a "utime" call to the
test file create app.  The utime command is suppose to set the mtime to
the current time with a NULL time parameter.  Same results though.  The
only way to get the "ls -l" (files sorted by name) and "ls -ltr" (sort
by time in reverse order) to produce the same results (i.e., files
listed in the order they were created)  is to put  a pause in the file
creation app.

Is there some UNIX/Solaris "feature" that would explain these results?
Any way to code the creation of the files without pausing in between
creating the files?

[ Post a follow-up to this message ]

"crashinc99@yahoo.com" <crashinc99@yahoo.com> writes:

> I'm trying to test the throughput of a data processing app by making
> numersous copies of the same test file (renamed with a one-up number)
> and measuring the time it takes to get the files through the system.
> I'm also trying to make sure files are being processed in order.
>
> The files  (DATA.0001.xml, DATA.0002.xml ... DATA.1000.xml) are placed
> in a directory which the app is polling.  I'm getting unexpected
> results with regard to the order the files are processed.  The app is
> polling the directory and grabbing all the "stable" files, they are
> sorted by the file's mtime (if the mtime is the same for 2 files, the
> filename is used).
>
> In checking the directory with 1000 test files, an ls -ltr (sort by
> time) produces unexpected results.  I'd expect the files to be listed
> in the order they were created (i.e., consistent with the one-up
> number), but this is not
> the case.

Please, write 100 times:

Unix file systems don't store creation time!
Unix file systems don't store creation time!
Unix file systems don't store creation time!
Unix file systems don't store creation time!
..
Unix file systems don't store creation time!

> I did some
> digging and it seems the "time last modified" on a file is not
> initialized to the creation time.  So, I added a "utime" call to the
> test file create app.  The utime command is suppose to set the mtime to
> the current time with a NULL time parameter.  Same results though.  The
> only way to get the "ls -l" (files sorted by name) and "ls -ltr" (sort
> by time in reverse order) to produce the same results (i.e., files
> listed in the order they were created)  is to put  a pause in the file
> creation app.

Moreover, the times that are stored are kept only with a one-second
resolution. Since current hardware can easily touch thousands of files
per second, you cannot distinguish files by time: USE THE NAME!


> Is there some UNIX/Solaris "feature" that would explain these results?
> Any way to code the creation of the files without pausing in between
> creating the files?

Well the date command gives a one-second resolution too, so it won't
be easy to get the exact creation time.

But one can easily use $(date +%Y%m%d) in the file name.

i=0; while [ $i -le 1000 ] ; do
touch file-$(date +%Y%m%d)-$(printf "%04d" $i) ; done


Really, I wonder hy you don't let the shell or ls sort the files by name?

i=0; while [ $i -le 1000 ] ; do touch file-$(printf "%04d" $i) ; done
ls
# or:
echo *

Now, if your program scans the directory itself, (using readdir), it
can get the names in any order, because the names are not sorted in
the directory.  With the above commands, it's ls or the shell who do
sort the file names.  Your application must do the same.

--
__Pascal Bourguignon__                     http://www.informatimago.com/
Small brave carnivores
Kill pine cones and mosquitoes
Fear vacuum cleaner

[ Post a follow-up to this message ]

Pascal Bourguignon wrote:
> Moreover, the times that are stored are kept only with a one-second
> resolution. Since current hardware can easily touch thousands of files
> per second, you cannot distinguish files by time: USE THE NAME!

Actually some platforms store a better resolution. Solaris, for
instance, stores nanosecond resolution in the stat struct and then hides
it for compatibility via macros. I suspect they're not really
nanoseconds but (microseconds * 1000) but in any case a program that
cares to can dig out the better resolution. I believe ls does so,
allowing it to correctly order files which appear to have been created
simultaneously.

HT

[ Post a follow-up to this message ]