|
Home > Archive > Unix Programming > May 2006 > Large file size and performance
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Large file size and performance
|
|
| nin234@yahoo.com 2006-05-18, 1:16 pm |
| I have an application (for trading stocks) that is a transaction
processing system, sending out orders using a socket messaging
protocol. It logs the messages to a file. The log file gets very large
as the day progresses to about 100 Mb or more. Will this large file
size cause a performance issue, in terms of sending out and receiving
socket messages. I am interested in understanding this at a linux
system level as to what happens. My application runs on Solaris and
Linux, are the performance impacting factors different on Solaris and
linux.
| |
|
| On 18 May 2006 10:43:40 -0700
"nin234@yahoo.com" <nin234@yahoo.com> wrote:
> I have an application (for trading stocks) that is a transaction
> processing system, sending out orders using a socket messaging
> protocol. It logs the messages to a file. The log file gets very large
> as the day progresses to about 100 Mb or more. Will this large file
> size cause a performance issue, in terms of sending out and receiving
> socket messages. I am interested in understanding this at a linux
> system level as to what happens. My application runs on Solaris and
> Linux, are the performance impacting factors different on Solaris and
> linux.
Shouldn't make any odds for sequential writes. Here's a test program I
write *just* for *you*. Makes a 37meg log file on my system without any
real degradation.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
void logline( char *text ) {
FILE *fp = fopen( "./logfile", "a+" );
if( !fp ) {
fprintf( stderr, "Cannot open logfile for append\n" );
exit( EXIT_FAILURE );
}
fprintf( fp, "%s\n", text );
fflush( fp );
fclose( fp );
}
int main() {
char *logtext=(char *)malloc(1024);
register int i;
int size;
time_t timer;
for( i=0 ; i<26 ; i++ ) {
logtext[i] = 'a'+i;
logtext[i+26] = 'A'+i;
}
timer = time( NULL );
for( i=0 ; i<655350 ; i++ ) {
logline( logtext );
if( i % 20000 == 0 ) {
fprintf( stdout, "%d seconds %i lines\n", \
time( NULL ) - timer, i );
}
}
free( logtext );
return( EXIT_SUCCESS );
}
--
Regards, Ed :: http://www.usenix.org.uk
proud bash person
:%s/Open Source/Free Software/g :: Free DNS available
| |
|
| On Thu, 18 May 2006 18:29:45 GMT
ed <ed@noreply.com> wrote:
> }
> fprintf( fp, "%s\n", text );
> fflush( fp );
Could try this too:
#include <unistd.h>
fdatasync( fp->_fileno );
> fclose( fp );
> }
Again I didn't notice any drop. If you think about it, all that's going
on is the file gets opened, the size is read and the pointer is assigned
the file size, then write happens, then the pointer is freed.
--
Regards, Ed :: http://www.sexeh.net
just another c++ hacker
:%s/Open Source/Free Software/g :: Free DNS available
|
|
|
|
|