|
Home > Archive > BizTalk Server General > October 2004 > Buffering flat-file messages
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Buffering flat-file messages
|
|
|
| I need to make sure a whole flat-file is disassembled before any of
its message parts begin an orchestration. What is the best way to go
about this?
| |
| Ruslan Yakushev [MSFT] 2004-10-22, 5:49 pm |
| If you use the standard Flat file disassembler component in the receive
pipeline and do not have any custom components in that pipeline then the
server will not publish any of the disassemlbed messages until the whole
flat file is disassmbled. Note however that this may not be enforced if the
transactional receive adapters are used (e.g. SQL). If you use FILE or HTTP
adapters it should work.
--------------------
>From: andy.berman@gmail.com (Andy)
>Newsgroups: microsoft.public.biztalk.general
>Subject: Buffering flat-file messages
>Date: 18 Oct 2004 00:06:28 -0700
>Organization: http://groups.google.com
>Lines: 3
>Message-ID: <9947a588.0410172306.6b54481@posting.google.com>
>NNTP-Posting-Host: 210.215.4.249
>Content-Type: text/plain; charset=ISO-8859-1
>Content-Transfer-Encoding: 8bit
>X-Trace: posting.google.com 1098083189 19412 127.0.0.1 (18 Oct 2004
07:06:29 GMT)
>X-Complaints-To: groups-abuse@google.com
>NNTP-Posting-Date: Mon, 18 Oct 2004 07:06:29 +0000 (UTC)
>Path:
cpmsftngxa06.phx.gbl!TK2MSFTNGP08.phx.gbl!newsfeed00.sul.t-online.de!t-onlin
e.de!news.glorb.com!postnews1.google.com!not-for-mail
>Xref: cpmsftngxa06.phx.gbl microsoft.public.biztalk.general:20607
>X-Tomcat-NG: microsoft.public.biztalk.general
>
>I need to make sure a whole flat-file is disassembled before any of
>its message parts begin an orchestration. What is the best way to go
>about this?
>
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
Note: For the benefit of the community-at-large, all responses to this
message are best directed to the newsgroup/thread from which they
originated.
| |
|
| Thank Ruslan,
that's promising news, however my pipeline needs to use a custom
disassembler so that certain flat-file lines can be stored in a sql
database, and later accessed when the first orchestration begins. If
my custom component uses the FFDasmComp base, does this mean the whole
file will be buffered?
My component currently works as it stores each flat-file line in
memory (based on the TransactionalFFDasm sample
http://msdn.microsoft.com/library/d...g_pipe_eiji.asp),
but I realise this will become a problem for large files. The only
other solution I could come up with is to store each message in
another database table as XML, but I'm struggling to work out how to
serialize an IBaseMessage.Context object.
Thanks Andy.
ruslany@online.microsoft.com (Ruslan Yakushev [MSFT]) wrote in message news:<DH$#i9HuEHA.764@cpmsftngxa10.phx.gbl>...
> If you use the standard Flat file disassembler component in the receive
> pipeline and do not have any custom components in that pipeline then the
> server will not publish any of the disassemlbed messages until the whole
> flat file is disassmbled. Note however that this may not be enforced if the
> transactional receive adapters are used (e.g. SQL). If you use FILE or HTTP
> adapters it should work.
>
>
> --------------------
> 07:06:29 GMT)
> cpmsftngxa06.phx.gbl!TK2MSFTNGP08.phx.gbl!newsfeed00.sul.t-online.de!t-onlin
> e.de!news.glorb.com!postnews1.google.com!not-for-mail
>
> This posting is provided "AS IS" with no warranties, and confers no rights.
> Use of included script samples are subject to the terms specified at
> http://www.microsoft.com/info/cpyright.htm
>
> Note: For the benefit of the community-at-large, all responses to this
> message are best directed to the newsgroup/thread from which they
> originated.
| |
| Ruslan Yakushev [MSFT] 2004-10-29, 5:49 pm |
| Using the TransactionalFFDasm is a good idea. This will ensure that you get
transactional behavior regardless of what adapter is used.
As I understood you have modified the MSDN sample to read data of each
message you get from base.GetNext() into a MemoryStream. You are right that
it will become an issue if you are using large messages. What you can do to
mitigate this issue is to save the message data into a VirtualStream
instead. VirtualStream is a stream that holds the data in memory up to
certain threshold and the rest of data it keeps on disk in temporary files.
You can find the VirtualStream implementation in the
ArbitraryXPathPropertyHandler SDK sample from the SDK refresh.
--------------------
>From: andy.berman@gmail.com (Andy)
>Newsgroups: microsoft.public.biztalk.general
>Subject: Re: Buffering flat-file messages
>Date: 23 Oct 2004 23:10:19 -0700
>Organization: http://groups.google.com
>Lines: 60
>Message-ID: <9947a588.0410232210.4af61d8f@posting.google.com>
>References: <9947a588.0410172306.6b54481@posting.google.com>
<DH$#i9HuEHA.764@cpmsftngxa10.phx.gbl>
>NNTP-Posting-Host: 210.215.4.249
>Content-Type: text/plain; charset=ISO-8859-1
>Content-Transfer-Encoding: 8bit
>X-Trace: posting.google.com 1098598219 20174 127.0.0.1 (24 Oct 2004
06:10:19 GMT)
>X-Complaints-To: groups-abuse@google.com
>NNTP-Posting-Date: Sun, 24 Oct 2004 06:10:19 +0000 (UTC)
>Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA03.phx.gbl!TK2MSFTNGP08.phx.gbl!news-out.cwi
x.com!newsfeed.cwix.com!news.maxwell.syr.edu!postnews1.google.com!not-for-ma
il
>Xref: cpmsftngxa10.phx.gbl microsoft.public.biztalk.general:20968
>X-Tomcat-NG: microsoft.public.biztalk.general
>
>Thank Ruslan,
>
>that's promising news, however my pipeline needs to use a custom
>disassembler so that certain flat-file lines can be stored in a sql
>database, and later accessed when the first orchestration begins. If
>my custom component uses the FFDasmComp base, does this mean the whole
>file will be buffered?
>
>My component currently works as it stores each flat-file line in
>memory (based on the TransactionalFFDasm sample
>http://msdn.microsoft.com/library/d...n-us/sdk/htm/eb
iz_prog_pipe_eiji.asp),
>but I realise this will become a problem for large files. The only
>other solution I could come up with is to store each message in
>another database table as XML, but I'm struggling to work out how to
>serialize an IBaseMessage.Context object.
>
>Thanks Andy.
>
>ruslany@online.microsoft.com (Ruslan Yakushev [MSFT]) wrote in message
news:<DH$#i9HuEHA.764@cpmsftngxa10.phx.gbl>...
the[vbcol=seagreen]
HTTP[vbcol=seagreen]
cpmsftngxa06.phx.gbl!TK2MSFTNGP08.phx.gbl!newsfeed00.sul.t-online.de!t-onlin[vbcol=seagreen]
rights.[vbcol=seagreen]
>
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm
Note: For the benefit of the community-at-large, all responses to this
message are best directed to the newsgroup/thread from which they
originated.
|
|
|
|
|