I have run in to a problem while attempting to transform a large-ish file
(50MB) in an orchestration.
The orchestration contains a simple Receive-->Transform-->Send  flow.

According to the orchestration debugger when procesing an example 15mb
inbound file on my development machine the time in the receive shape is
negligible, the transform takes approximately 5 minutes and the send takes
20 minutes(??)

The larger 50MB file will not process because it is constantly thrashing
virtual memory.

Is there some setting in the send or the construct message shapes which
could be causing these processes to take large chunks of memory?

The transformation is a fairly straightforward re-ordering of fields where 1
record in the source creates 1 record in the target, so I doubt it has
anything to do with the transformation complexity.

regards,

Tim C.

[ Post a follow-up to this message ]

Transformation is doing XSLT.
XSLT is a great technology but requires the engine to take the whole
document in memory and build it up in the XML DOM!!! So believe me,
transforming a 50Mb file is not a very good practice... XSLT is just not
suiteable for those kind of things...  Actually this is not a problem
related to BizTalk Server, but rather to the way the XSLT works.

Using:
-  a custom .NET component
-  the XmlReader classes
-  the XmlWriter classes
would be a better idea.

Best regards,
Christof


"Microsoft User" <microsoft@javelingroup.com> wrote in message
news:%23fKcPeGWEHA.2996@TK2MSFTNGP12.phx.gbl...
> I have run in to a problem while attempting to transform a large-ish file
> (50MB) in an orchestration.
> The orchestration contains a simple Receive-->Transform-->Send  flow.
>
> According to the orchestration debugger when procesing an example 15mb
> inbound file on my development machine the time in the receive shape is
> negligible, the transform takes approximately 5 minutes and the send takes
> 20 minutes(??)
>
> The larger 50MB file will not process because it is constantly thrashing
> virtual memory.
>
> Is there some setting in the send or the construct message shapes which
> could be causing these processes to take large chunks of memory?
>
> The transformation is a fairly straightforward re-ordering of fields where
1
> record in the source creates 1 record in the target, so I doubt it has
> anything to do with the transformation complexity.
>
> regards,
>
> Tim C.
>
>

[ Post a follow-up to this message ]

Ok, I assumed that when the BTS 2004 features list suggested support for
large files that this extended as far as the mapping functionality.

I have an input text file and an output text file of a different format,
would it not be more sensible to go with a single intermediate schema and
write a custom flat file disassembler on the inbound pipeline and a custom
assembler on the outbound pipeline. (I already have a custom disassembler in
place to make the inbound file viable for parsing to a sensible schema).
Or in fact do everything in the inbound disassembler.

The only remaining annoyance is then having to code the database lookups to
replace the Database functoids from the map.

regards,

Tim C.

"Christof" <bts2004communityfeedback (at) hotmail (dot) com>
wrote in message news:uxO0pTIWEHA.3492@TK2MSFTNGP10.phx.gbl...
> Transformation is doing XSLT.
> XSLT is a great technology but requires the engine to take the whole
> document in memory and build it up in the XML DOM!!! So believe me,
> transforming a 50Mb file is not a very good practice... XSLT is just not
> suiteable for those kind of things...  Actually this is not a problem
> related to BizTalk Server, but rather to the way the XSLT works.
>
> Using:
>  -  a custom .NET component
>  -  the XmlReader classes
>  -  the XmlWriter classes
> would be a better idea.
>
> Best regards,
> Christof
>
>
> "Microsoft User" <microsoft@javelingroup.com> wrote in message
> news:%23fKcPeGWEHA.2996@TK2MSFTNGP12.phx.gbl... 
file[vbcol=seagreen] 
takes[vbcol=seagreen] 
where[vbcol=seagreen]
> 1 
>
>

[ Post a follow-up to this message ]