|
Home > Archive > BizTalk Server General > January 2006 > Flat File problem
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
|
|
| Brian Cesafsky 2006-01-20, 5:55 pm |
| I have a process that accepts a flat file that has many records in it. The
record is a 10 digit phone number followed by a CRLF. I have an
orchestration that reads the flat file input and simple writes the record(s)
to a SQL server DB. Everyting is working great when I have under 5000
records in it, but as I get larger and larger in my tests, BTS is not
processing the file. It takes about 14 minutes to process the 5000 records.
My issue is that for production processing, I will have in excess of 500,000
records to process in the input file. The input file is not huge (just 5
MB, but the record count is very high)
Why can't BTS handle this, and what do you have for suggestion(s)???
Thanks in advance!
Brian
| |
|
| Are you processing through a soap adapter? If so, there is a KB article on
the subject: Q886966.
You will need to set the following registry values to values that will be
determined by number of records you are reading. There is a formula in the
article that you can use. The registry settings are these:
- MaxCompletionPortThreads (which might be able to be left alone, 220).
- MinWorkerThreads (this is the number of threads that each processor will
create - Maximum number of messages that you expect to load on the SOAP
adapter + 10%).
- MaxWorkerThreads (not sure -- I was advised it to can be set to 2000 when
I was processing a file of 2000 - 500K seems too huge)
Restart after you change the values.
You will need to read the article and experiment with setting values.
Good luck,
Mo
"Brian Cesafsky" wrote:
> I have a process that accepts a flat file that has many records in it. The
> record is a 10 digit phone number followed by a CRLF. I have an
> orchestration that reads the flat file input and simple writes the record(s)
> to a SQL server DB. Everyting is working great when I have under 5000
> records in it, but as I get larger and larger in my tests, BTS is not
> processing the file. It takes about 14 minutes to process the 5000 records.
>
>
> My issue is that for production processing, I will have in excess of 500,000
> records to process in the input file. The input file is not huge (just 5
> MB, but the record count is very high)
>
>
> Why can't BTS handle this, and what do you have for suggestion(s)???
>
>
> Thanks in advance!
>
> Brian
>
>
>
| |
| BizWorld 2006-01-23, 2:53 am |
| i am also facing the same problem, in my case, file size is 70 MB and there
are 350000 records. when i tried to process whole file, it never come back.
then i wrote some .net code to split the file into chunks of 2 MB each and
process the file. now process took 6 hours which is not acceptable. it is
very disappointing for developers as this is a very basic scenario that any
one can face. especially the health care providers file are very large and
they are facing issues of large file processing.
Regards,
BizWorld
"Brian Cesafsky" <brian.cesafsky@autotrackerplus.com> wrote in message
news:u52ksfgHGHA.3100@tk2msftngp13.phx.gbl...
>I have a process that accepts a flat file that has many records in it. The
>record is a 10 digit phone number followed by a CRLF. I have an
>orchestration that reads the flat file input and simple writes the
>record(s) to a SQL server DB. Everyting is working great when I have under
>5000 records in it, but as I get larger and larger in my tests, BTS is not
>processing the file. It takes about 14 minutes to process the 5000
>records.
>
>
> My issue is that for production processing, I will have in excess of
> 500,000 records to process in the input file. The input file is not huge
> (just 5 MB, but the record count is very high)
>
>
> Why can't BTS handle this, and what do you have for suggestion(s)???
>
>
> Thanks in advance!
>
> Brian
>
>
| |
| Brian Cesafsky 2006-01-23, 6:02 pm |
| I'm not using a SOAP adapter, I am just processing from a network folder, so
I'm just using a File Adapter. What solved my problem was simply setting
the Max Occurs to unbounded in my schema for the input file. I believe that
the entire file will then be read as one message and not 500,000 seperate
messages that each need to be inserted in the DB. I found a good article on
how to split up a file into batches... here is the link
http://geekswithblogs.net/sthomas/a...2/12/17373.aspx
and also this one...
http://www.biztalkgurus.com/Samples...r-Pipeline.html
Can anyone recommend if I should use batches or just do it as I am right now
by just setting the Max Occurs to unbounded.
"Brian Cesafsky" <brian.cesafsky@autotrackerplus.com> wrote in message
news:u52ksfgHGHA.3100@tk2msftngp13.phx.gbl...
>I have a process that accepts a flat file that has many records in it. The
>record is a 10 digit phone number followed by a CRLF. I have an
>orchestration that reads the flat file input and simple writes the
>record(s) to a SQL server DB. Everyting is working great when I have under
>5000 records in it, but as I get larger and larger in my tests, BTS is not
>processing the file. It takes about 14 minutes to process the 5000
>records.
>
>
> My issue is that for production processing, I will have in excess of
> 500,000 records to process in the input file. The input file is not huge
> (just 5 MB, but the record count is very high)
>
>
> Why can't BTS handle this, and what do you have for suggestion(s)???
>
>
> Thanks in advance!
>
> Brian
>
>
|
|
|
|
|