|
Home > Archive > BizTalk Server > January 2007 > 200MB and bigger flat files
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
200MB and bigger flat files
|
|
| Sinisa Catic 2006-12-19, 1:19 pm |
| Hi, I am doing some proof of concept tests to find best solution to process
flat files that are very large (from BizTalk perspective): 200MB and bigger.
One solution is to split file into smaller chunks in the receive pipeline by
inserting carriage return (before its converted to XML). Right now I am doing
such test with the 100MB flat file that will be split into 29 pieces each
with the size around 3.5MB 9I am gradualy increasing file size). I am doing
tests with various batch sizes and have found that bigger batch size
generally cuts processing time. I am doing splitting only and routing
outgoing XML into separate send folder. As I have expected processing is slow
for such large input file but I wonder why is not faster because I have
noticed that CPU is almost not used after file is taken and split into
chunks. Memory also stays flat all the time (1.4GB from the total of 2GB, 4
logical processors). It looks processing is IO bound and not CPU or memory
bound. I wonder if processing could be shortened by adding more messagebox
databases or maybe tuning some parameters. It does not look even that SQL
Server is bottleneck. More looks like the problem is how BizTalk is using SQL
Server. Any suggestion would be welcome. Thanks.
Sinisa Catic
| |
| Paul Somers 2007-01-09, 1:32 am |
| Are you using BizTalk 2006?
I have been able to process 200 mb flat files through BizTalk 2006
successfully, it did take a bit of time I recall, I did need to ensure that
I had sufficient memory on the BizTalk server, even though with 2006 it
caches things to disk.
"Sinisa Catic" <SinisaCatic@discussions.microsoft.com> wrote in message
news:CEEBC2FC-9115-4672-B615-81CFE60DBE3B@microsoft.com...
> Hi, I am doing some proof of concept tests to find best solution to
> process
> flat files that are very large (from BizTalk perspective): 200MB and
> bigger.
> One solution is to split file into smaller chunks in the receive pipeline
> by
> inserting carriage return (before its converted to XML). Right now I am
> doing
> such test with the 100MB flat file that will be split into 29 pieces each
> with the size around 3.5MB 9I am gradualy increasing file size). I am
> doing
> tests with various batch sizes and have found that bigger batch size
> generally cuts processing time. I am doing splitting only and routing
> outgoing XML into separate send folder. As I have expected processing is
> slow
> for such large input file but I wonder why is not faster because I have
> noticed that CPU is almost not used after file is taken and split into
> chunks. Memory also stays flat all the time (1.4GB from the total of 2GB,
> 4
> logical processors). It looks processing is IO bound and not CPU or memory
> bound. I wonder if processing could be shortened by adding more messagebox
> databases or maybe tuning some parameters. It does not look even that SQL
> Server is bottleneck. More looks like the problem is how BizTalk is using
> SQL
> Server. Any suggestion would be welcome. Thanks.
>
> Sinisa Catic
|
|
|
|
|