I have a process that accepts a flat file that has many records in it.  The
record is a 10 digit phone number followed by a CRLF.  I have an
orchestration that reads the flat file input and simple writes the record(s)
to a SQL server DB.  Everyting is working great when I have under 5000
records in it, but as I get larger and larger in my tests, BTS is not
processing the file.  It takes about 14 minutes to process the 5000 records.


My issue is that for production processing, I will have in excess of 500,000
records to process in the input file.  The input file is not huge (just 5
MB, but the record count is very high)


Why can't BTS handle this, and what do you have for suggestion(s)???


Thanks in advance!

Brian

[ Post a follow-up to this message ]

Are you processing through a soap adapter? If so, there is a KB article on
the subject: Q886966.

You will need to set the following registry values to values that will be
determined by number of records you are reading. There is a formula in the
article that you can use. The registry settings are these:
- MaxCompletionPortThreads (which might be able to be left alone, 220).
- MinWorkerThreads (this is the number of threads that each processor will
create - Maximum number of messages that you expect to load on the SOAP
adapter + 10%).
- MaxWorkerThreads (not sure -- I was advised it to can be set to 2000 when
I was processing a file of 2000 - 500K seems too huge)

Restart after you change the values.

You will need to read the article and experiment with setting values.

Good luck,
Mo

"Brian Cesafsky" wrote:

> I have a process that accepts a flat file that has many records in it.  Th
e
> record is a 10 digit phone number followed by a CRLF.  I have an
> orchestration that reads the flat file input and simple writes the record(
s)
> to a SQL server DB.  Everyting is working great when I have under 5000
> records in it, but as I get larger and larger in my tests, BTS is not
> processing the file.  It takes about 14 minutes to process the 5000 record
s.
>
>
> My issue is that for production processing, I will have in excess of 500,0
00
> records to process in the input file.  The input file is not huge (just 5
> MB, but the record count is very high)
>
>
> Why can't BTS handle this, and what do you have for suggestion(s)???
>
>
> Thanks in advance!
>
> Brian
>
>
>

[ Post a follow-up to this message ]

i am also facing the same problem, in my case, file size is 70 MB and there
are 350000 records. when i tried to process whole file, it never come back.
then i wrote some .net code to split the file into chunks of 2 MB each and
process the file. now process took 6 hours which is not acceptable. it is
very disappointing for developers as this is a very basic scenario that any
one can face. especially the health care providers file are very large and
they are facing issues of large file processing.

Regards,
BizWorld


"Brian Cesafsky" <brian.cesafsky@autotrackerplus.com> wrote in message
news:u52ksfgHGHA.3100@tk2msftngp13.phx.gbl...
>I have a process that accepts a flat file that has many records in it.  The
>record is a 10 digit phone number followed by a CRLF.  I have an
>orchestration that reads the flat file input and simple writes the
>record(s) to a SQL server DB.  Everyting is working great when I have under
>5000 records in it, but as I get larger and larger in my tests, BTS is not
>processing the file.  It takes about 14 minutes to process the 5000
>records.
>
>
> My issue is that for production processing, I will have in excess of
> 500,000 records to process in the input file.  The input file is not huge
> (just 5 MB, but the record count is very high)
>
>
> Why can't BTS handle this, and what do you have for suggestion(s)???
>
>
> Thanks in advance!
>
> Brian
>
>

[ Post a follow-up to this message ]

I'm not using a SOAP adapter, I am just processing from a network folder, so
I'm just using a File Adapter.  What solved my problem was simply setting
the Max Occurs to unbounded in my schema for the input file.  I believe that
the entire file will then be read as one message and not 500,000 seperate
messages that each need to be inserted in the DB.  I found a good article on
how to split up a file into batches... here is the link

http://geekswithblogs.net/sthomas/a...2/12/17373.aspx

and also this one...

http://www.biztalkgurus.com/Samples...r-Pipeline.html


Can anyone recommend if I should use batches or just do it as I am right now
by just setting the Max Occurs to unbounded.
"Brian Cesafsky" <brian.cesafsky@autotrackerplus.com> wrote in message
news:u52ksfgHGHA.3100@tk2msftngp13.phx.gbl...
>I have a process that accepts a flat file that has many records in it.  The
>record is a 10 digit phone number followed by a CRLF.  I have an
>orchestration that reads the flat file input and simple writes the
>record(s) to a SQL server DB.  Everyting is working great when I have under
>5000 records in it, but as I get larger and larger in my tests, BTS is not
>processing the file.  It takes about 14 minutes to process the 5000
>records.
>
>
> My issue is that for production processing, I will have in excess of
> 500,000 records to process in the input file.  The input file is not huge
> (just 5 MB, but the record count is very high)
>
>
> Why can't BTS handle this, and what do you have for suggestion(s)???
>
>
> Thanks in advance!
>
> Brian
>
>

[ Post a follow-up to this message ]