|
Home > Archive > BizTalk Server General > October 2004 > Orchestration Best-Practice Advice Needed
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Orchestration Best-Practice Advice Needed
|
|
|
| Hi,
I'm building a workflow that at business-level begins Purchase Order, then
Shipment , Goods Receipt and Invoice. It's typical of many BizTalk 2004
Implementations, I'd imagine. My proposal is to encompass the entire
"order-to-invoice" flow into a single long-running orchestration that calls
"sub" orchestrations which handle the specifics of each business transaction.
So for example, when my main orchestration receives a Purchase Order Message
(activate receive launches a new orchestration instance), it calls the PO
orchestration which has the PO message as one of its parameters.
In system terms, the flow runs rather like this (some details missed-out):
1. XML PO received by BTS 2004 (as file)
2. XML PO passed by BTS to Webservice which creates a new PO in our ERP system
3. ERP (via XML WServer) returns response plus details of order created
4. BTS creates an order acknowledgement message which goes back to customer.
5. After a long wait (days), the ERP system ships the goods. BTS (via SQL
adapter) retrieves details of shipment from ERP system and creates Shipment
message which goes to customer.
6. Some time later, upon physical receipt of goods by customer, a
Goods-Receipt message is received by BTS (XML message file).
7. BTS sends GR message to ERP via webservice.
8. After another long wait (days), ERP generates invoice. BTS retrieves
invoice data from ERP (via SQL adapter), creates invoice message and sends it
to customer.
There's nothing technically inventive about the above approach, but I've got
a couple of questions that I'd appreciate feedback on:
(1) There might be 300+ simultaneous instances, and the lifetime of each
instance might run up to a couple of weeks. Is it normal to handle an entire
order-invoice flow like this? Would it be better to split each 'message' into
completely separate orchestrations rather than bind them together within a
calling orchestration? I really would rather avoid doing this though, as we
want to use BizTalk as a workflow tool, rather than just as a message
generator. Also some data in the PO message is not interesting to our ERP
system but is required within subsequent shipment/invoice messages, and so
will be mapped by BTS from message-to-message only, rather than being passed
to our ERP.
(2) I assume I need to use correlation sets/types to manage the correlation
between a specific Purchase Order Instance and Shipment (say). But how can I
best handle the situation whereby (for whatever reason) we don't receive a PO
via BTS, but we still want to send a Shipment Message via BTS? The shipment
message data comes straight from the ERP (I realise that some data might be
missing from the Shipment message, as mentioned in (1)). I really don't want
to split the "calling" orchestration into separate per-message
orchestrations. Can the "calling" orchestration be built in such a way that
it could be activated from the shipment stage?
Advice from anyone more experienced than me in long-running processes would
be much appreciated.
Many thanks!
| |
| Greg Forsythe 2004-10-04, 5:52 pm |
| This is exactly what Biztalk is designed to do.
(1) 300+ simultaneous instances is relatively minor
Biztalk has been designed to handle a whole order-invoice process like this.
It will give you visibility into the process, using BAM/BAS you can analyse
the process by stage, by customer, by product...
(2)You can create a single orchestration with multiple activate receives in
a listen shape. A listen only processes one branch unlike the parallel
action that must process all branches
- very rough outline below
Listen
__________|____________
| |
receivePO(activate) receiveERPShipment(activate)
sendPO-ERP |
sendOrderAck |
receiveERPShipment(correlate) |
createShipmentMessage createShipmentMessage
|_____________________ |
|
sendShipment
|
receiveGoodsreceipt
|
sendGR-ERP
|
receiveInvoice
|
sendInvoice
You will need the Biztalk compiler to create a convoy to prevent activating
a new orchestration as well as matching an existing orchestration instance
i.e. sending 2 Shipment messages. This could take a bit of trial and error -
there is no create convoy flag.
Doing this in separate orchestrations would be even trickier and require
assistance from the ERP.
hope this is helpful
Greg
"GAT" <GAT@discussions.microsoft.com> wrote in message
news:0422FDDF-D686-4567-AD91-8CDAAD08F08D@microsoft.com...
> Hi,
>
> I'm building a workflow that at business-level begins Purchase Order, then
> Shipment , Goods Receipt and Invoice. It's typical of many BizTalk 2004
> Implementations, I'd imagine. My proposal is to encompass the entire
> "order-to-invoice" flow into a single long-running orchestration that
calls
> "sub" orchestrations which handle the specifics of each business
transaction.
> So for example, when my main orchestration receives a Purchase Order
Message
> (activate receive launches a new orchestration instance), it calls the PO
> orchestration which has the PO message as one of its parameters.
>
> In system terms, the flow runs rather like this (some details missed-out):
>
> 1. XML PO received by BTS 2004 (as file)
> 2. XML PO passed by BTS to Webservice which creates a new PO in our ERP
system
> 3. ERP (via XML WServer) returns response plus details of order created
> 4. BTS creates an order acknowledgement message which goes back to
customer.
> 5. After a long wait (days), the ERP system ships the goods. BTS (via SQL
> adapter) retrieves details of shipment from ERP system and creates
Shipment
> message which goes to customer.
> 6. Some time later, upon physical receipt of goods by customer, a
> Goods-Receipt message is received by BTS (XML message file).
> 7. BTS sends GR message to ERP via webservice.
> 8. After another long wait (days), ERP generates invoice. BTS retrieves
> invoice data from ERP (via SQL adapter), creates invoice message and sends
it
> to customer.
>
> There's nothing technically inventive about the above approach, but I've
got
> a couple of questions that I'd appreciate feedback on:
>
> (1) There might be 300+ simultaneous instances, and the lifetime of each
> instance might run up to a couple of weeks. Is it normal to handle an
entire
> order-invoice flow like this? Would it be better to split each 'message'
into
> completely separate orchestrations rather than bind them together within a
> calling orchestration? I really would rather avoid doing this though, as
we
> want to use BizTalk as a workflow tool, rather than just as a message
> generator. Also some data in the PO message is not interesting to our ERP
> system but is required within subsequent shipment/invoice messages, and so
> will be mapped by BTS from message-to-message only, rather than being
passed
> to our ERP.
>
> (2) I assume I need to use correlation sets/types to manage the
correlation
> between a specific Purchase Order Instance and Shipment (say). But how can
I
> best handle the situation whereby (for whatever reason) we don't receive a
PO
> via BTS, but we still want to send a Shipment Message via BTS? The
shipment
> message data comes straight from the ERP (I realise that some data might
be
> missing from the Shipment message, as mentioned in (1)). I really don't
want
> to split the "calling" orchestration into separate per-message
> orchestrations. Can the "calling" orchestration be built in such a way
that
> it could be activated from the shipment stage?
>
> Advice from anyone more experienced than me in long-running processes
would
> be much appreciated.
>
> Many thanks!
| |
|
| Greg, many thanks for your help. Sorry it's been so long, I have been away.
I tried the solution that you prescribe. The problem is exactly as you
predict: in the event of receipt of a PO, two Shipment messages are sent.
This is because (referring to Greg's diagram), the
ReceiveERPShipment(correlate) and (activate) receive shapes are connected to
the same file-receive port. So in this case, the correlated instance
processes the shipment message, but so does a new orchestration instance. To
set this scenario up, I configured the sendPO-ERP shape to initiate the
correlation set, and the receiveERPShipment(correlate) shape to follow it.
So, I tried another approach: forcing a sequential correlated receive
(convoy) scenario. To illustrate my problem a bit clearer, imagine I need to
implement an orchestration that goes like this:
Receive Message 1 (activate, initiate correlation)
¦
Send Modified Message 1
¦
Receive Message 2 (correlate)
¦
Send Modified Message 2
¦
Receive Message 3 (correlate)
¦
Send Modified Message 3
¦
End
All messages use the File port. Assume that Modified Messages 2 and 3
contain elements mapped from Messages 1 and 2 (ie. it's a workflow, not a
discrete process). I need to build my orchestration so it will still run if
(for whatever reason) Message 1 or Message 2 is not received. So for example,
if a Message 3 is received for which there is no initiated correlation set,
it is processed from the point of "Receive Message 3". Of course, some
elements in "Modified Message 3" wouldn't be mapped, but that's OK for this
illustration.
Greg, your answer got so close to a solution. But when I built it like this,
I had problems:
Listen
¦
----------------------------------------------------------------
¦ ¦ ¦
Rec Message 1 ¦ ¦
(activate, init corr) Rec Message 2 (act, init. Cor) Rec Message
3 (act)
¦ ¦ ¦
Send Mod. Msg 1 Send Mod. Msg 2 Send Mod. Msg 3
¦ ¦ ¦
Rec. Msg 2 (corr) Rec Msg 3 (corr) ¦
¦ ¦ ¦
Send Mod. Msg 2 Send Mod. Msg 3 ¦
¦ ¦ ¦
Rec. Msg 3 (corr) ¦ ¦
¦ ¦ ¦
Send Mod. Msg 3 ¦ ¦
¦ ¦ ¦
----------------------------------------------------------------
¦
(if you copy/paste this into notepad, it should look OK)
It's the same correlation type used throughout. The problem is that on
compilation Biztalk detects a sequential correlated receive (convoy) and
insists that the Message type should be the same for the activate receive
shapes (top row of listen shape). Each of these receive shapes are connected
to the same receive port, but a different operation within that port (each
having a different message type).
So if I change the model so the correlation sets are initiated by the first
send shape in each branch, it works but I get multiple instances of the
orchestration launched (one for the correlated instance, plus one or more of
the other branches). Hum!
So, what's the best thing to do? Does anyone know? Can I modify this design
in some way, or is there a completely different way of solving the
requirement. As always, help would be appreciated.
Thanks!
| |
| Glynn Timmins 2004-10-19, 7:47 am |
| Please, help! Anyone!
I'm in a real fix with this problem, and no-one seems to know an
answer... I have changed the receive port so its Ordered Delivery
property is true (it's bound to a file receive port). So, now the
activate-receive shape at the top of each branch initiates a
correlation set (the same set), and subsequent receive shapes follow
that set. The send shapes don't follow a c-set. But again, the same
behaviour... I get multiple instances of the orchestration: one for
the correlated message, and another one or two depending on which
message I submit. So if I send a message1, followed by a message2 and
a message3, I get three instances. If I send a message2 followed by a
message3, I get two instances. You can see why this is: each receive
is linked to the same port.
It seemed like such a 'standard' requirement, but the solution seems
so hard-to-get! I don't know if it's because I'm using the wrong
technique to solve the problem, or if it's just something slight that
I've missed. But surely there must be a way to design an orchestration
so it can cope with missing messages?
Please help.... anyone.
Thanks!
| |
| Greg Forsythe 2004-10-25, 8:46 pm |
| Glynn, sorry about the delay, I have also been away.
The problem here is managing the pub-sub mechanism on the MessageBox
database.
The way correlation works is by using this pub-sub mechanism - basically the
orchestration subscribes to a message, setting filters based on the
correlation set
For the shipment message, say you are correlating on a PONumber field then
the receiveERPShipment(correlate) will subscribe to shipment messages where
the PONumber field exists and has a value equal to that set when the
correlation was initialised. This subscription is transitory only exists
while the receiveERPShipment(correlate) is able to receive
Effective filters:
BTS.MessageType = Shipment
AND Shipment.PONumber exists
AND Shipment.PONumber = 12345
The receiveERPShipment(activate) just subscribes to shipment messages.
Effective filters:
BTS.MessageType = Shipment
A correlated shipment message will satisfy both sets of filters and both
receives will be given the shipment message.
In the case of an uncorrelated shipment message only the
receiveERPShipment(activate) will be given the message, as there will be no
other subscriptions that match.
When convoys are created the subscription has an extra attribute (can only
set by the compiler) that specifies a convoy id. If the subscription has a
convoy id, then all subscriptions with the same convoy id are checked and
only the best match receives the message. Apprently setting a convoy with
multiple activate receives is very difficult if not impossible to set up.
The easiest way to solve this problem is to expand the filters on the
receiveERPShipment(activate) so that it will not match a correlated shipment
message. This will require a field in the message that is unique or has a
unique value in a correlated shipment message - the unique value may be an
empty string
Is there some field that you could use to filter the correlated shipment
messages.
regards
Greg
"Glynn Timmins" <glynnatimmins@hotmail.com> wrote in message
news:5228d043.0410190201.7f692a0@posting.google.com...
> Please, help! Anyone!
>
> I'm in a real fix with this problem, and no-one seems to know an
> answer... I have changed the receive port so its Ordered Delivery
> property is true (it's bound to a file receive port). So, now the
> activate-receive shape at the top of each branch initiates a
> correlation set (the same set), and subsequent receive shapes follow
> that set. The send shapes don't follow a c-set. But again, the same
> behaviour... I get multiple instances of the orchestration: one for
> the correlated message, and another one or two depending on which
> message I submit. So if I send a message1, followed by a message2 and
> a message3, I get three instances. If I send a message2 followed by a
> message3, I get two instances. You can see why this is: each receive
> is linked to the same port.
>
> It seemed like such a 'standard' requirement, but the solution seems
> so hard-to-get! I don't know if it's because I'm using the wrong
> technique to solve the problem, or if it's just something slight that
> I've missed. But surely there must be a way to design an orchestration
> so it can cope with missing messages?
>
> Please help.... anyone.
>
> Thanks!
| |
|
| Greg, you're a star. Thanks for your detailed explanation, which led to my
belated flash of inspiration. There's no reliable candidate fields which
could be used to distinguish non-subscribed from subscribed messages.
Therefore, my proposal is for Biztalk to maintain an "audit" table within the
ERP. So, when the customer sends a PO, Biztalk processes it, and writes a
record to the audit table (where the primary key is the PO_Num). When the
Shipment data is retrieved from the ERP via the SQL Adapter, it will have an
additional boolean field "Correlated", which indicates the presence of a
related record in the audit table. It's that field I'll filter within the
orchestration.
Again, thanks very much, you've certainly relieved one major headache!
|
|
|
|
|