BizTalk Server General - Sequential Convoy Orchestration and High Instance State Messages

Author

Sequential Convoy Orchestration and High Instance State Messages

Craig Vermeer

2006-02-26, 10:47 am

How's that for a title? :-D

Sorry for the cross-post, but I'm hoping to find a solution to this quickly.

I have a sequential convoy orchestration that was built with the goal of
throttling output to a certain vendor. Basically, messages are
correlated by MessageType to the Orchestration instance, which loops
through incoming messages and sends them to a send port with Delivery
Notification turned on. This forces a FIFO behavior in the orchestration.

I'm seeing some performance problems under production load. What
happened was that the vendor receiving the messages had some performance
problems for a period of time, causing a large backlog of messages to
build up in the orchestration. Now it's the orchestration itself that
is performing slowly.

What I'm pretty sure is causing the problem is that the orchestration is
dehydrating and re-hydrating itself with each message. Because there
are 30k instance state messages associated with the Orchestration, it is
dehydrating and rehydrating a huge amount of data each time. When
viewing the Orchestration through Orchestration Debugger, I see a > 30
second gap when just leaving a scope shape. This makes sense, because
from what I've heard scope shapes can trigger dehydration.

I have this orchestration being fed through an MSMQt queue. Is there
any way I can set things up so that the messages stay in the queue until
the Orchestration asks for it via a Receive shape? Is it just a
property I need to set on the receive port in the Orchestration?

Thanks in advance,
Craig

Tomas Restrepo \(MVP\)

2006-02-26, 10:47 am

Hi Craig,

I don't know all the answers (sorry) but fwiw, here are a few comments
inline...

> I have a sequential convoy orchestration that was built with the goal of
> throttling output to a certain vendor. Basically, messages are correlated
> by MessageType to the Orchestration instance, which loops through incoming
> messages and sends them to a send port with Delivery Notification turned
> on. This forces a FIFO behavior in the orchestration.
>
> I'm seeing some performance problems under production load. What happened
> was that the vendor receiving the messages had some performance problems
> for a period of time, causing a large backlog of messages to build up in
> the orchestration. Now it's the orchestration itself that is performing
> slowly.
>
> What I'm pretty sure is causing the problem is that the orchestration is
> dehydrating and re-hydrating itself with each message. Because there are
> 30k instance state messages associated with the Orchestration, it is
> dehydrating and rehydrating a huge amount of data each time. When viewing
> the Orchestration through Orchestration Debugger, I see a > 30 second gap
> when just leaving a scope shape. This makes sense, because from what I've
> heard scope shapes can trigger dehydration.

it might just be persistence points, and those definitely involve
serializing state to the db (at least as I view things, dehydration implies
a persistence point, but not necessarily otherwise, I might be wrong).

> I have this orchestration being fed through an MSMQt queue. Is there any
> way I can set things up so that the messages stay in the queue until the
> Orchestration asks for it via a Receive shape?

Not really, since the receive port simply takes messages (almost as fast as
it can, depending on available resources) and pushes them into the
messagebox. As long as the messages can be routed, the messagebox is happy
to take them and correlate them to the active subscription of your
orchestration's receive shape.

It sounds to me like you're going to need to change the design somewhat to
have accumulate messages in another way, so that biztalk doesn't choke on
them. The usual workaround I've seen for performance problems with convoys
of this nature is to have the messages simple routed into a SQL database
that records the order of the message, and then have an orchestration that
reads them "manually" and sends them.

--
Tomas Restrepo
tomasr@mvps.org
http://www.winterdom.com/

Craig Vermeer

2006-02-26, 10:47 am

Hrm. The SQL solution would definitely work.

What about this? Since the goal of my Orchestration is throttling and
not real FIFO behavior, what if I correlated the messages on something
other than MessageType?

Say, for example, I promoted a property to the Message Context that
consisted of the the current time, rounded to the nearest minute. If I
correlated on MessageType and this other property, I would get a new
Orchestration instance every minute. Each Orchestration instance would
process its batch of messages in a FIFO manner. While this would result
in a sort of batching functionality rather than the one at a time
functionality I currently have, I'm sure that this would still be
acceptable.

Tomas Restrepo (MVP) wrote:
> Hi Craig,
>
> I don't know all the answers (sorry) but fwiw, here are a few comments
> inline...
>
>
> it might just be persistence points, and those definitely involve
> serializing state to the db (at least as I view things, dehydration implies
> a persistence point, but not necessarily otherwise, I might be wrong).
>
>
> Not really, since the receive port simply takes messages (almost as fast as
> it can, depending on available resources) and pushes them into the
> messagebox. As long as the messages can be routed, the messagebox is happy
> to take them and correlate them to the active subscription of your
> orchestration's receive shape.
>
> It sounds to me like you're going to need to change the design somewhat to
> have accumulate messages in another way, so that biztalk doesn't choke on
> them. The usual workaround I've seen for performance problems with convoys
> of this nature is to have the messages simple routed into a SQL database
> that records the order of the message, and then have an orchestration that
> reads them "manually" and sends them.
>
>