|
Home > Archive > BizTalk Server General > February 2004 > How to Handle Large Files?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
How to Handle Large Files?
|
|
| Petrucci2000 2004-02-08, 8:37 am |
| Hi All, We are not using XLANG!
I am having a requirement in which a large file possibly 10MB is being sent
to us. We are using MSMQ Receive Function. Since the MSMQ size limit is only
4Mb I want to figure out how can I process this size files?
Our sender is willing to send this file in chunks of 4MB as a part of
transaction! will that help in processing ? The MSMQ recv function is tied 2
a preprocessor which has 2 further process the files.
Alt solution is to implement a custom component which will read 4mb chunks
out of MSMQ and write 2 disk and then some file recv function will pick it
up from ther.
TIA
| |
| Jan Eliasen 2004-02-08, 8:37 am |
| On Mon, 5 Jan 2004 12:19:11 -0500, "Petrucci2000"
<nospam@microsoft.com> wrote:
If you really want to use MSMQ for the transport then you will
probably need to write your own application that takes the chunks of
data and merges them into one document and submits this document to
BizTalk.
>Hi All, We are not using XLANG!
>I am having a requirement in which a large file possibly 10MB is being sent
>to us. We are using MSMQ Receive Function. Since the MSMQ size limit is only
>4Mb I want to figure out how can I process this size files?
>Our sender is willing to send this file in chunks of 4MB as a part of
>transaction! will that help in processing ? The MSMQ recv function is tied 2
>a preprocessor which has 2 further process the files.
>
>Alt solution is to implement a custom component which will read 4mb chunks
>out of MSMQ and write 2 disk and then some file recv function will pick it
>up from ther.
>
>TIA
>
>
>
--
Jan Eliasen, representing himself and not the company he works for
| |
| Nick Malik 2004-02-08, 8:37 am |
| It won't help to write a custom component to read chunks out of MSMQ,
because the size limit is 4mb for an MSMQ message. Therefore, if someone
submits a 10mb file to MSMQ, it will simply fail, and your component will
never be called.
You need to write a custom component that will work on the SENDING side to
break up the data file into chunks, and then add the chunks to the MSMQ
queue. Biztalk would just read these chunks normally.
If this is not reasonable, then you should probably consider using some
other form of transport other than MSMQ.
Like Secure FTP.
Hope this helps,
-- Nick
"Petrucci2000" <nospam@microsoft.com> wrote in message
news:efbD8960DHA.3140@tk2msftngp13.phx.gbl...
> Hi All, We are not using XLANG!
> I am having a requirement in which a large file possibly 10MB is being
sent
> to us. We are using MSMQ Receive Function. Since the MSMQ size limit is
only
> 4Mb I want to figure out how can I process this size files?
> Our sender is willing to send this file in chunks of 4MB as a part of
> transaction! will that help in processing ? The MSMQ recv function is tied
2
> a preprocessor which has 2 further process the files.
>
> Alt solution is to implement a custom component which will read 4mb chunks
> out of MSMQ and write 2 disk and then some file recv function will pick it
> up from ther.
>
> TIA
>
>
>
>
| |
| Jan Eliasen 2004-02-08, 8:37 am |
| On Tue, 06 Jan 2004 07:58:02 GMT, "Nick Malik"
<nickmalik@hotmail.nospam.com> wrote:
Do you mean that BizTalk can read a document that consists of many 4MB
chunks? How does it know when the document ends?
>It won't help to write a custom component to read chunks out of MSMQ,
>because the size limit is 4mb for an MSMQ message. Therefore, if someone
>submits a 10mb file to MSMQ, it will simply fail, and your component will
>never be called.
>
>You need to write a custom component that will work on the SENDING side to
>break up the data file into chunks, and then add the chunks to the MSMQ
>queue. Biztalk would just read these chunks normally.
>
>If this is not reasonable, then you should probably consider using some
>other form of transport other than MSMQ.
>
>Like Secure FTP.
>
>Hope this helps,
>-- Nick
>
>"Petrucci2000" <nospam@microsoft.com> wrote in message
>news:efbD8960DHA.3140@tk2msftngp13.phx.gbl...
>sent
>only
>2
>
--
Jan Eliasen, representing himself and not the company he works for
| |
| Nick Malik 2004-02-08, 8:37 am |
| Hi Jan,
This depends on the nature of the transactions. If a document is
self-consistent and represents a single transaction, it may be difficult
(not impossible) to break apart. Then again, it is rare to see a single
transaction of this size. It is probably a feed that contains many hundreds
or thousands of individual records. These kinds of files tend to get quite
large, but are usually easy to break into chunks.
I have files of this kind coming into a healthcare payer from a series of
healthcare providers. The files can easily range to 15MB. In this case,
we are not using MSMQ for transport since the trading partner is not within
the local network.
The original question suggested that he was looking for a solution where the
file could be broken up, so I was assuming that the nature of his data
allows for a reasonable way to create "chunks."
At no time should someone send a 10MB file, representing a single
transaction, into Biztalk. The preprocessor that the questioner uses should
create smaller transactions for submitting to Biztalk. Otherwise, the
schema should be noted so that the parser is able to break up the file into
multiple documents for Biztalk to process. Otherwise, a file of that size
would probably choke Biztalk.
HTH,
--- Nick
"Jan Eliasen" <spam@eliasen.dk> wrote in message
news:dg3lvvg1his95cj2124p44e7um1fhu4n3h@
4ax.com...[color=blue]
> On Tue, 06 Jan 2004 07:58:02 GMT, "Nick Malik"
> <nickmalik@hotmail.nospam.com> wrote:
>
> Do you mean that BizTalk can read a document that consists of many 4MB
> chunks? How does it know when the document ends?
>
to[color=blue]
| |
|
|
Check out www.itemfield.com
>-----Original Message-----
>On Mon, 5 Jan 2004 12:19:11 -0500, "Petrucci2000"
><nospam@microsoft.com> wrote:
>
>If you really want to use MSMQ for the transport then
you will
>probably need to write your own application that takes
the chunks of
>data and merges them into one document and submits this
document to
>BizTalk.
>
possibly 10MB is being sent[color=blue]
MSMQ size limit is only[color=blue]
files?[color=blue]
4MB as a part of[color=blue]
recv function is tied 2[color=blue]
will read 4mb chunks[color=blue]
function will pick it[color=blue]
>
>--
>Jan Eliasen, representing himself and not the company he
works for
>.
>
|
|
|
|
|