BizTalk Server Orchestration - Random Message Subscription Error

Author

Random Message Subscription Error - It works then it don't.

Steve

2004-09-07, 7:48 am

We're scratching our heads in disbelief. We have a deployed
orchestration that uses a custom adapter to communicate with another
system at various stages throughout the orchestration. This
architecture has been working fine and I tested it successfully a
number of times. Then I tried to demo it to some colleagues (yes,
curse of the demo strikes again!). On the last communication on the
adapter the following errors were logged in the AppEvent log:

Event Type: Error
Event Source: BizTalk Server 2004
Event Category: BizTalk Server 2004
Event ID: 5778
Date: 02/09/2004
Time: 14:18:53
User: N/A
Computer: <<computername>>
Description:
The Messaging engine failed to process a message submitted by
adapter:<<adaptername>> Source URL:<<url>>. Details:Could not find a
matching subscription for the message. . This error occurs if the
subscribed orchestration schedule or send port has not been started,
or if some of the message properties necessary for subscription
evaluation have not been promoted. Please refer to Health and Activity
Tracking tool for more detailed information on this failure

.... and ...

Event Type: Error
Event Source: BizTalk Server 2004
Event Category: BizTalk Server 2004
Event ID: 5753
Date: 02/09/2004
Time: 14:18:53
User: N/A
Computer: <<computername>>
Description:
The "<<adaptername>>" adapter is suspending a message coming from
Source URL:"<<url>>". Details:"Unknown Error Description ".

In HAT, a routing failure report was logged. It appears that the
error was raised at the Receive Shape that is waiting for a response
message via our custom adapter. We checked the subscription
information via the SubscriptionViewer and the response message looks
fine (ie has the correct promoted properties, has the correct receive
port id etc).

I then attempted to run another process through the same orchestration
from the start - repeating the exact steps as I did in the demo 15
mins earlier - and it all worked fine, progressing to the end of the
orchestration successfully. Since then we have run many tests and
every once in a while the message fails to reach BizTalk via the
adapter.

Has anyone experienced anything like this? There were no system
updates, service restarts or reboots throughout the duration of the
test scenarios. At our best guess, it seems when the process fails
that maybe the subscription information was not ready by the time the
response message was received. Is this technically
possible/impossible? Are there any other possiblilities that someone
can suggest for what went wrong?

After our development headaches it finally seemed that our BizTalk
integration was stable and predictable - but now this has occurred and
our confidence in BizTalk has taken a hit.

Lee Graber [MSFT]

2004-09-07, 5:53 pm

Look in HAT. Are you getting lots of Completed With Discarded Messages
orchestrations? When you are communicating via the adapter, are you using
delivery notification or solicit/response sendports and so letting us
handle the correlation or are you using custom correlation sets? If you are
using custom correlation sets, when you run your tests, do you make sure
you use unique correlation values, or are you just sending the same doc
through, over and over, with the same correlation information inside.
Correlation sets must be unique, If you test a system with non-unique
correlation sets, then you will get messages routing all over the place and
your results will be a lot of Completed With Discarded Messages and a bunch
of routing faliures. Is this what you are doing?

HTH
Lee

This posting is provided "AS IS" with no warranties, and confers no rights.

EBusiness Server Team
--------------------[vbcol=seagreen]
11:08:50 GMT)[vbcol=seagreen]
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFTNGP08.phx.gbl!news-out.cwi
x.com!newsfeed.cwix.com!newsfeed.icl.net!newsfeed.fjserv.net!news.tele.dk!ne
ws.tele.dk!small.news.tele.dk!proxad.net!postnews2.google.com!not-for-mail[vbcol=seagreen]

Steve

2004-09-08, 2:47 am

Lee,

Thanks for your response. Our communication is asynchronous as
opposed to solicit/response. We define a correlation set based on 5
values in the message. 2 of the 5 values are guaranteed to be unique
for each communication that occurs during our test scenario, including
a timestamp and a string identifier describing the type of
communication. The other 3 values in the correlation type relate to
the customer name (string), a service name (string) and a GUID which
identifies the current customer-service instance (we call this the
caseId). For the test scenarios I'm running the customer and process
names are the same for each case. Furthermore, I only run a single
case through the orchestration at any time during the test.

We are not getting Completed with Discarded Messages in HAT - only
Routing Failure Reports where the status is Suspended (Not Resumable),
Error Code -1 and Error Category -3 for what it's worth. For each
Routing Failure Report in HAT there is a corresponding Messaging
Service Class which says that the adapter "is suspending a message
coming from the Source URL:<<url>> Details: 'Unknown Error
Description'." Again the status of this message is Suspended (Not
Resumable) and, if it helps, the Error Code is 0x80004005 and Error
Category is 0.

We receive this error across both our development environments and our
test environment at a rate of approximately 10-20% (ie 80-90% of
communications succeed without any problems).

I really appreciate your help and am hoping you can continue to steer
us toward a solution. If there is any additional information you
need, please let me know.

Thanks in advance.

Steve

leegr@microsoft.com (Lee Graber [MSFT]) wrote in message news:<uCFhj5OlEHA.3608@cpmsftngxa10.phx.gbl>...[vbcol=seagreen]
> Look in HAT. Are you getting lots of Completed With Discarded Messages
> orchestrations? When you are communicating via the adapter, are you using
> delivery notification or solicit/response sendports and so letting us
> handle the correlation or are you using custom correlation sets? If you are
> using custom correlation sets, when you run your tests, do you make sure
> you use unique correlation values, or are you just sending the same doc
> through, over and over, with the same correlation information inside.
> Correlation sets must be unique, If you test a system with non-unique
> correlation sets, then you will get messages routing all over the place and
> your results will be a lot of Completed With Discarded Messages and a bunch
> of routing faliures. Is this what you are doing?
>
> HTH
> Lee
>
> This posting is provided "AS IS" with no warranties, and confers no rights.
>
> EBusiness Server Team
> --------------------
> 11:08:50 GMT)
> cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFTNGP08.phx.gbl!news-out.cwi
> x.com!newsfeed.cwix.com!newsfeed.icl.net!newsfeed.fjserv.net!news.tele.dk!ne
> ws.tele.dk!small.news.tele.dk!proxad.net!postnews2.google.com!not-for-mail

Lee Graber [MSFT]

2004-09-08, 5:49 pm

So while I am confused about the error associated with the suspended
message from the isolated host (ie why it is not "no subscription"), the
fact is still that you got a routing failure. I actually just posted
something about this on my blog (http://blogs.msdn.com/Biztalk_Core_Engine)
and in the end, it will fall into one of the categories of cases I
described there. To see if you are duplicating your correlation sets, you
could try to track the correlation properties (just use HAT's configuration
menu to mark the properties of the schema as tracked. Then after you have
hit a message which fails to route, you can search in HAT (Find Message
View) for a message of that schema type with that correaltion set values
(the guid should be enough). That would tell you if it went through
already. If it did not already go through, then you must have one of the
other race conditions I discussed in my blog entry no zombies. Is it always
the same message that fails to route (I don't know how many messages you
having flowing around). If so, is this message part of a parallel listen
shape, ie is there a way for the orchestration to finish without this
message actually being received. If that is possible, then you are most
likely hitting the race condition but just not getting zombies because of
how far apart the timing is. Based on your scenario, you should use one of
the two approaches I described here to debug further.

HTH
Lee

This posting is provided "AS IS" with no warranties, and confers no rights.

EBusiness Server Team
--------------------[vbcol=seagreen]
<uCFhj5OlEHA.3608@cpmsftngxa10.phx.gbl>[vbcol=seagreen]
08:30:48 GMT)[vbcol=seagreen]
cpmsftngxa10.phx.gbl!TK2MSFTNGXS01.phx.gbl!cpmsftngxa06.phx.gbl!TK2MSFTNGP08
.phx.gbl!newsfeed00.sul.t-online.de!t-online.de!news.glorb.com!postnews2.goo
gle.com!not-for-mail[vbcol=seagreen]
news:<uCFhj5OlEHA.3608@cpmsftngxa10.phx.gbl>...[vbcol=seagreen]
using[vbcol=seagreen]
are[vbcol=seagreen]
sure[vbcol=seagreen]
and[vbcol=seagreen]
bunch[vbcol=seagreen]
rights.[vbcol=seagreen]
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFTNGP08.phx.gbl!news-out.cwi[vbcol=seagreen]
x.com!newsfeed.cwix.com!newsfeed.icl.net!newsfeed.fjserv.net!news.tele.dk!ne[vbcol=seagreen]
ws.tele.dk!small.news.tele.dk!proxad.net!postnews2.google.com!not-for-mail[vbcol=seagreen]

Joep Wijers

2004-09-09, 7:47 am

Hi Lee,

I am Joep, a collegue of Steve, and I am the one who developed the
adapter.
I have some details and corrections that may be relevant for a good
understanding of our problem.

you wrote:
> So while I am confused about the error associated with the suspended
> message from the isolated host (ie why it is not "no subscription"), the
> fact is still that you got a routing failure.

- our adapter runs in the default host, not in the isolated host.
- we do get an event log entry saying that there is no subscription
- however, we can see that the subsription exists and that the 5
properties in the correlation set match exactly:
- the Subscription Viewer shows us the subscription and the 5
expected values
- the HAT shows us the values of these 5 properties in the message
context
that was preserved in the routing failure.
- in an earlier stage we had the same error due to a sitiation in
which there were multiple ports for the same address, which caused the
response to arrive at the wrong port. This is now no longer the case,
but it still occ

you also wrote:
>I actually just posted
> something about this on my blog (http://blogs.msdn.com/Biztalk_Core_Engine)
> and in the end, it will fall into one of the categories of cases I
> described there. To see if you are duplicating your correlation sets, you
> could try to track the correlation properties (just use HAT's configuration
> menu to mark the properties of the schema as tracked. Then after you have
> hit a message which fails to route, you can search in HAT (Find Message
> View) for a message of that schema type with that correaltion set values
> (the guid should be enough). That would tell you if it went through
> already.
This is very interesting. But in this case I would think that the fact
that the subscription viewer still shows the matching subscription
after the routing failure occurred, proofs that the message was not
already sent (otherwise the subscription would be gone, wouldn't it?).

you also wrote:
>If it did not already go through, then you must have one of the
> other race conditions I discussed in my blog entry no zombies. Is it always
> the same message that fails to route (I don't know how many messages you
> having flowing around).
>If so, is this message part of a parallel listen
> shape, ie is there a way for the orchestration to finish without this
> message actually being received. If that is possible, then you are most
> likely hitting the race condition but just not getting zombies because of
> how far apart the timing is. Based on your scenario, you should use one of
> the two approaches I described here to debug further.

After the routing failure, the orchestration is still running and
waiting for the message to arrive.
But I will read your blog entry on race conditions!

Thanks,
Joep

david2005

2005-04-21, 2:48 am

Hi,

I have developed a simlar custom adapter with Adapter framwork and
gettings exactly same kind of error message (randomly) in the production
enviorment. does any one have got solution for the above problem . its
urgent. my confifence on biztalk is getting very low

Thanks in adavnace

david2005

2005-04-21, 2:48 am

Steve

2005-04-22, 5:54 pm

David,

We ended up abandoning communication via our custom adapter and
instead developed a generic web service listener and wait
orchestration that our customer orchestrations can use to communicate
with our software. After this rewrite we have experienced no lost
communications. It's a little messier but the main thing is it works.

Steve

"david2005" <david2005@nospam.yahoo.com> wrote in message news:< f7d0c7d9456db770bc9af9e9e1441173@localho
st.talkaboutsoftware.com>...
> Hi,
>
> I have developed a simlar custom adapter with Adapter framwork and
> gettings exactly same kind of error message (randomly) in the production
> enviorment. does any one have got solution for the above problem . its
> urgent. my confifence on biztalk is getting very low
>
> Thanks in adavnace