BizTalk Server - BTS 2002 SP1, performance bottleneck in Flat File serializer

Author

BTS 2002 SP1, performance bottleneck in Flat File serializer

Henrik Sandqvist

2004-09-30, 10:40 am

Hi
I've run into what looks like a performance/throughput bottleneck in the
Flat File Serializer component in BTS 2002 when it's generating files with
large number of records (25000+) in them. My inbound document is an XML-file
with invoicing data (thus the size) and outbound document is a positional
flat file needed by our legacy system. A small map is also applied in the
channel.

Example:
An inbound document with 27.000 records takes 4 minutes to submit into
BizTalk and the resulting flat file is completed after another 27 minutes
(!!!) in the work queue. When I remove the envelope spec from the Port
definition (hence eliminating the Serializer but still using the map and
creating an XML-file as output) and rerun the XML-file the submission time is
the same but I get my resulting file in less than 2 minutes (15 times faster
than with the serializer).

My only conclusion is that the Flat File Serializer seems to have a serious
performance/throughput problem and it also seems to be linear i.e. breaking
up the files into smaller parts will not speed up the total process or
resource consumption.

The system used in the above test has a single 2,4GHz P-IV CPU with 512 MB
of RAM, BTS 2002 SP1. During the work queue processing BizTalk server
consumes 100% of the CPU but there is plenty of RAM available. No other
messages were processed by the server during testing.

Has anybody experienced something like this? Is there an explanation (maybe
from Microsoft)?
We're targeting 200-400.000 records/day and letting the files sit in BizTalk
spending CPU-hours (!) is not an attractive solution. Writing a custom AIC
just to produce a flat file is not a very good option either, in my opinion
this should work in the shipped standard component with acceptable
performance, I don't demand a lightning fast solution but definitely one
better than this.

Regards
/Henrik Sandqvist
Resco AB

Joe Eldridge \(MSFT\)

2004-10-05, 5:54 pm

How large is the inbound flatfile?

--
Joe Eldridge [MSFT]
This posting is provided "AS IS", with no warranties, and confers no rights.
"Henrik Sandqvist" <HenrikSandqvist@discussions.microsoft.com> wrote in
message news:75F50EBF-8492-44ED-AD25-3777C0DD9284@microsoft.com...
> Hi
> I've run into what looks like a performance/throughput bottleneck in the
> Flat File Serializer component in BTS 2002 when it's generating files with
> large number of records (25000+) in them. My inbound document is an
> XML-file
> with invoicing data (thus the size) and outbound document is a positional
> flat file needed by our legacy system. A small map is also applied in the
> channel.
>
> Example:
> An inbound document with 27.000 records takes 4 minutes to submit into
> BizTalk and the resulting flat file is completed after another 27 minutes
> (!!!) in the work queue. When I remove the envelope spec from the Port
> definition (hence eliminating the Serializer but still using the map and
> creating an XML-file as output) and rerun the XML-file the submission time
> is
> the same but I get my resulting file in less than 2 minutes (15 times
> faster
> than with the serializer).
>
> My only conclusion is that the Flat File Serializer seems to have a
> serious
> performance/throughput problem and it also seems to be linear i.e.
> breaking
> up the files into smaller parts will not speed up the total process or
> resource consumption.
>
> The system used in the above test has a single 2,4GHz P-IV CPU with 512 MB
> of RAM, BTS 2002 SP1. During the work queue processing BizTalk server
> consumes 100% of the CPU but there is plenty of RAM available. No other
> messages were processed by the server during testing.
>
> Has anybody experienced something like this? Is there an explanation
> (maybe
> from Microsoft)?
> We're targeting 200-400.000 records/day and letting the files sit in
> BizTalk
> spending CPU-hours (!) is not an attractive solution. Writing a custom AIC
> just to produce a flat file is not a very good option either, in my
> opinion
> this should work in the shipped standard component with acceptable
> performance, I don't demand a lightning fast solution but definitely one
> better than this.
>
> Regards
> /Henrik Sandqvist
> Resco AB

Henrik Sandqvist

2004-10-06, 2:47 am

There is no inbound flatfile... the inbound document is XML (an ADO.NET
dataset) and the outbound document is a positional flatfile i.e. my problem
is the opposite of the more common problem with too large inbound flatfiles.
Message sizes and tracking is currently not a problem.
My problem occurs when the Flatfile Envelope in the Messaging Port is
applied and invokes the BizTalk Flat File Serializer that actually creates
the flat file.

I've found a workaround that is pretty ugly but it tells us a bit of what's
going on.
The outbound flatfile specification has a total of 19 fields (total length
80). If I change the specification to have only 1 field with length 80 and
then let the map concatenate all the inbound XML-fields into on large string
that is mapped to this only field then the file is completed in less than 2,5
minutes, almost as fast as running without the envelope.
It seems to me that the flat file serializer problem is related to the
number of fields in the specification not the size of the message.
The drawback is that this workaround is disabling all the functionality and
control that is supplied in the message/envelope specification such as data
validation, formatting and positioning since everything has to be done in
scripts in the map.
Regards
/Henrik Sandqvist

"Joe Eldridge (MSFT)" wrote:

> How large is the inbound flatfile?
>
> --
> Joe Eldridge [MSFT]
> This posting is provided "AS IS", with no warranties, and confers no rights.
> "Henrik Sandqvist" <HenrikSandqvist@discussions.microsoft.com> wrote in
> message news:75F50EBF-8492-44ED-AD25-3777C0DD9284@microsoft.com...
>
>
>