|
Home > Archive > BizTalk Server General > October 2004 > What is an advantage of using pipeline component?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
What is an advantage of using pipeline component?
|
|
| Alex Star 2004-10-15, 9:11 pm |
| Hello,
Currently I have working a simple BizTalk 2004 solution with flat-file
incoming and EDI outgoing. So, it is 2 schemas, 1 map, and very simple
orchestration and it works just fine!
Recently I receive request to edit incoming file before it converted to EDI,
so I need to rip some lines of records and instead of transferring this info
to EDI - just to save it to DB for the record.
Here is my original incoming file example:
Version ORIGINAL
RECORD1 PART1 ############
RECORD1 PART2 ############
RECORD2 PART1 ############
RECORD2 PART2 ############
RECORD3 PART1 ############
RECORD3 PART2 ############
RECORD4 PART1 ############
RECORD4 PART2 ############
Here what I need:
Version CORRECTED
RECORD1 PART1 ############
RECORD1 PART2 ############
RECORD4 PART1 ############
RECORD4 PART2 ############
Version LEFTOWER
RECORD2 PART1 ############
RECORD2 PART2 ############
RECORD3 PART1 ############
RECORD3 PART2 ############
As I can see, the best solution for it is just create a custom pipeline
component and convert incoming file to 2 files, change value of "VERSION"
header, promote it and use as "Decision criteria" at "Decide shape" and
convert it or just save an information to DB. It is a really simple solution.
However, I just wondering, why I have to create a pipeline component? What
is advantage of it? As I understand, I can’t return from it more them one
message at the time and I will need to drop "LEFTOWER" part to incoming
folder directly.
Is it not easy to just build a stand-alone component (not part of BizTalk
enterprise) and simply divide this file and drop both files to the same
place? In this case I need just slightly change my schema (promote version
property) and change orchestration.
So, what do you think?
Alex Star
astar@NOSPAMsumitrans.com
| |
| Gilles [MSFT] 2004-10-18, 5:49 pm |
| Hello,
>Currently I have working a simple BizTalk 2004 solution with flat-file
>incoming and EDI outgoing. So, it is 2 schemas, 1 map, and very simple
>orchestration and it works just fine!
>Recently I receive request to edit incoming file before it converted to EDI,
>so I need to rip some lines of records and instead of transferring this info
>to EDI - just to save it to DB for the record.
>
>Here is my original incoming file example:
>
>Version ORIGINAL
>RECORD1 PART1 ############
>RECORD1 PART2 ############
>RECORD2 PART1 ############
>RECORD2 PART2 ############
>RECORD3 PART1 ############
>RECORD3 PART2 ############
>RECORD4 PART1 ############
>RECORD4 PART2 ############
>
>Here what I need:
>
>Version CORRECTED
>RECORD1 PART1 ############
>RECORD1 PART2 ############
>RECORD4 PART1 ############
>RECORD4 PART2 ############
>
>Version LEFTOWER
>RECORD2 PART1 ############
>RECORD2 PART2 ############
>RECORD3 PART1 ############
>RECORD3 PART2 ############
>
>
>
>As I can see, the best solution for it is just create a custom pipeline
>component and convert incoming file to 2 files, change value of "VERSION"
>header, promote it and use as "Decision criteria" at "Decide shape" and
>convert it or just save an information to DB. It is a really simple solution.
>
>However, I just wondering, why I have to create a pipeline component? What
>is advantage of it? As I understand, I can’t return from it more them one
>message at the time and I will need to drop "LEFTOWER" part to incoming
>folder directly.
>Is it not easy to just build a stand-alone component (not part of BizTalk
>enterprise) and simply divide this file and drop both files to the same
>place?
You can indeed perform any processing outside of BizTalk.
BizTalk 2004 provides tracking for messages in pipeline components. This is something
you'd need to implement in your extrernal processing if you would like to take advantage of it. Moreover,
any error in parsing will be caught and saved by BizTalk messaging engine so you can debug
failures (and have a trace of them) a little more easily. You could of course build this capability into your
external processing but everytime you duplicate a BizTalk feature, you'd have to test it for correctness, performances
and scalability.
In a pipeline component, BizTalk offers the necessary streaming mecanisms so you can implement a
very scalable component (for instance, by not loading the whole message in memory but stream it through components instead).
You could indeed write a very scalable transformation externally from BizTalk and be just fine but how much work would that be?
Maybe you do not even need to process large messages.
Another thing you might want to consider. With a pipeline component, you get clustering for free. Just run the pipeline
in a host of multiple machines. Adding more processing power translates into a few clicks to re-configure the host which runs
the pipeline. A custom solution might not be as easy to load balance and not as easy to reconfigure when new hardware
is added/shiffled around.
If you implement your solution with an "external" piece, it might make the application harder to deploy because it is not possible
to take the "content" of BizTalk and move it to another machine anymore: external pieces must be moved as well.
One more thing to consider: future releases of BizTalk will problaby add new features to pipeline components. If you have a custom component,
you set yourself up for taking advantage of these new features without much work. For instance, currently, BAM does
not allow tracking in pipelines but a future version of BizTalk might extend it to allow full pipeline tracking. Of course, you could always re-write the
solution as a component later or maybe decide that you do not need to take advantage of new features.
BizTalk pipelines support encryption/signing. If in the future you are asked to deal with encrypted documents, you'll have to implement that
in your external solution. With BizTalk, you coul dput a decrypt pipeline component and be done with it.
With what you explain, it does not seem that there is a clear cut "better" solution. At the end of the day,
it will boil down to what is the fastest to implement, the most reliable and the easiest to maintain and extend.
Perhaps in your case, a simple perl/sed script (or some custom code) is easy to write and to maintain.
Personally, I feel more confident using the messaging support in BizTalk because I know it well and it has
received a decent amount of testing for performances and scalability. Third parties sell components I can re-use if needed
and they'll support me. Personally, I'd just go and implement the pipeline component. However, this is my personal point of view.
Thanks,
-Gilles.
| |
| Matt Milner 2004-10-19, 8:47 pm |
| Based on the requirements, it sounds like you have an incoming document and
you need to split it to two outgoing formats. To me this sounds like simple
mapping with some content based routing. Setup a send port group with a
subscription to your incoming message or to the receive port. next, add two
send ports to this group, one for the SQL and one for the EDI. Create two
maps, one from flat file to sql and one from flat file to EDI. Setup these
maps in the appropriate send ports.
This design is scalable because you can add send ports to the send port
group with their own maps and continue to grow the destinations you are
sending two without having to write code.
Did I miss some requirement that would keep this from working for you ?
Matt
"Gilles [MSFT]" <Gilles@online.microsoft.com> wrote in message
news:UBG1IXTtEHA.720@cpmsftngxa06.phx.gbl...
> Hello,
>
>
> You can indeed perform any processing outside of BizTalk.
> BizTalk 2004 provides tracking for messages in pipeline components. This
> is something
> you'd need to implement in your extrernal processing if you would like to
> take advantage of it. Moreover,
> any error in parsing will be caught and saved by BizTalk messaging engine
> so you can debug
> failures (and have a trace of them) a little more easily. You could of
> course build this capability into your
> external processing but everytime you duplicate a BizTalk feature, you'd
> have to test it for correctness, performances
> and scalability.
>
> In a pipeline component, BizTalk offers the necessary streaming mecanisms
> so you can implement a
> very scalable component (for instance, by not loading the whole message in
> memory but stream it through components instead).
> You could indeed write a very scalable transformation externally from
> BizTalk and be just fine but how much work would that be?
> Maybe you do not even need to process large messages.
>
> Another thing you might want to consider. With a pipeline component, you
> get clustering for free. Just run the pipeline
> in a host of multiple machines. Adding more processing power translates
> into a few clicks to re-configure the host which runs
> the pipeline. A custom solution might not be as easy to load balance and
> not as easy to reconfigure when new hardware
> is added/shiffled around.
>
> If you implement your solution with an "external" piece, it might make the
> application harder to deploy because it is not possible
> to take the "content" of BizTalk and move it to another machine anymore:
> external pieces must be moved as well.
>
> One more thing to consider: future releases of BizTalk will problaby add
> new features to pipeline components. If you have a custom component,
> you set yourself up for taking advantage of these new features without
> much work. For instance, currently, BAM does
> not allow tracking in pipelines but a future version of BizTalk might
> extend it to allow full pipeline tracking. Of course, you could always
> re-write the
> solution as a component later or maybe decide that you do not need to take
> advantage of new features.
>
> BizTalk pipelines support encryption/signing. If in the future you are
> asked to deal with encrypted documents, you'll have to implement that
> in your external solution. With BizTalk, you coul dput a decrypt pipeline
> component and be done with it.
>
> With what you explain, it does not seem that there is a clear cut "better"
> solution. At the end of the day,
> it will boil down to what is the fastest to implement, the most reliable
> and the easiest to maintain and extend.
> Perhaps in your case, a simple perl/sed script (or some custom code) is
> easy to write and to maintain.
>
> Personally, I feel more confident using the messaging support in BizTalk
> because I know it well and it has
> received a decent amount of testing for performances and scalability.
> Third parties sell components I can re-use if needed
> and they'll support me. Personally, I'd just go and implement the pipeline
> component. However, this is my personal point of view.
>
> Thanks,
> -Gilles.
>
| |
| Alex Star 2004-10-20, 5:48 pm |
| Hi Matt,
Thanks for this posting.
However, the problem is that I can not point both maps to the same
source-flat-file. It is not a problem to retrieve information from original
file and save it to DB. I actually do not have to create a second map for it
because I could implement this logic to an existing map.
The problem is- how can I exclude some records from the map-transformations?
Here is how it works now:
Left schema (source)
Root
----Loop
---------Record1
------------------Child1
------------------Child2
------------------Child3
------------------Child4
------------------Child5
................................
................................
---------Record2
------------------Child1
------------------Child2
------------------Child3
------------------Child4
------------------Child5
................................
................................
Each pair (Record1 and Record2) inside loop is converting to EDI and the
number or iterations of this loop is a variable and it could be 1 or 3000.
So, the problem is: How is it possible to exclude some pair from the
map-transformations? And because of this problem, I can't just create a
second map as you suggesting and assign it to the same source. This is why I
need to modify the source before it was processed by "Flat file to EDI" map.
Currently I completed this task outside of BizTalk and I can split this file
to 2 and post each of them to different incoming locations-one for EDI
transfer and another for DB update.
However, I agree with Gilles that it is duplicating of BizTalk
functionality, so my next step is - pipeline component.
I founded a really good example of "Unzip Pipeline component" together with
"Pipeline component wizard" here:
http://martijnh.blogspot.com/2004/0...ent-wizard.html
Best regards,
Alex Star.
| |
| Matt Milner 2004-10-21, 5:51 pm |
| What is the business rule for excluding records from the destination? I
would think that in most cases you could capture that with logical functoids
and looping in a map.
matt
"Alex Star" <AlexStar@discussions.microsoft.com> wrote in message
news:CB93C96A-14B8-4BFA-ACCB-457B6E5F1799@microsoft.com...
> Hi Matt,
> Thanks for this posting.
> However, the problem is that I can not point both maps to the same
> source-flat-file. It is not a problem to retrieve information from
> original
> file and save it to DB. I actually do not have to create a second map for
> it
> because I could implement this logic to an existing map.
> The problem is- how can I exclude some records from the
> map-transformations?
> Here is how it works now:
> Left schema (source)
> Root
> ----Loop
> ---------Record1
> ------------------Child1
> ------------------Child2
> ------------------Child3
> ------------------Child4
> ------------------Child5
> ...............................
> ...............................
>
> ---------Record2
> ------------------Child1
> ------------------Child2
> ------------------Child3
> ------------------Child4
> ------------------Child5
> ...............................
> ...............................
>
>
> Each pair (Record1 and Record2) inside loop is converting to EDI and the
> number or iterations of this loop is a variable and it could be 1 or 3000.
> So, the problem is: How is it possible to exclude some pair from the
> map-transformations? And because of this problem, I can't just create a
> second map as you suggesting and assign it to the same source. This is why
> I
> need to modify the source before it was processed by "Flat file to EDI"
> map.
>
> Currently I completed this task outside of BizTalk and I can split this
> file
> to 2 and post each of them to different incoming locations-one for EDI
> transfer and another for DB update.
> However, I agree with Gilles that it is duplicating of BizTalk
> functionality, so my next step is - pipeline component.
> I founded a really good example of "Unzip Pipeline component" together
> with
> "Pipeline component wizard" here:
> http://martijnh.blogspot.com/2004/0...ent-wizard.html
>
> Best regards,
>
> Alex Star.
>
>
| |
| Alex Star 2004-10-21, 5:51 pm |
| The business rule is really simple and easy to implement and it based on
simple value comparison. If the record is belongs to certain payable (My
solution is financial application to convert original flat-file to SAP BAPI
file) then this node should be excluded from transferring by map. In the map
level each iteration of "Source file's Record loop" will produce a
"Destination file" record and both loops has been connected in the map by
"looping" functoid, which is responsible for keeping consistent "Destination"
document (basically, the number of iterations on the left equal to the number
of iterations on the right).
On the map level we can't split message to separate messages.
Alex Star
| |
| Matt Milner 2004-10-21, 8:46 pm |
| I guess I must be missing something or not following well. It seems like a
combination of routing, mapping and orchestration logic would do this. It
sounds like you have a solution that works for you with the pipeline
component though.
Matt
"Alex Star" <AlexStar@discussions.microsoft.com> wrote in message
news:218E2289-D701-42BB-8303-67C51AF8AF21@microsoft.com...
> The business rule is really simple and easy to implement and it based on
> simple value comparison. If the record is belongs to certain payable (My
> solution is financial application to convert original flat-file to SAP
> BAPI
> file) then this node should be excluded from transferring by map. In the
> map
> level each iteration of "Source file's Record loop" will produce a
> "Destination file" record and both loops has been connected in the map by
> "looping" functoid, which is responsible for keeping consistent
> "Destination"
> document (basically, the number of iterations on the left equal to the
> number
> of iterations on the right).
> On the map level we can't split message to separate messages.
>
> Alex Star
>
|
|
|
|
|