Hello folks, I have a question regarding BizTalk 2004.  I have reviewed
most of the Microsoft website's PR material, but I'm having a hard time
making heads or tails of it, and BPI in general.

If I could, let me describe some scenarios for you.  If you could, could
you tell me if you think BizTalk 2004 would be a good solution for this or
not.  Thanks in advance.

My employer has an application which is deployed in most of the US states.
It is used to collect data for our client, which is stored in a SQL Server
database at each site.  Throughout the day, a task is run which exports
data from the local SQL Server (based upon specific criteria) and sends it
to a central repository at our client's main headquarters.  The data is
sent via FTP.

When HQ receives the data, it performs some processing, validates the data,
etc, and ultimately loads it into a SQL Server database.  An
acknowledgement file is sent to the client site (actually it is placed in
the FTP directory and the client downloads it).

We are about to being a project to create significant enhancements to the
uploading service, including porting the code to the .NET framework.
Additional enhancements include supporting different "types" of data
transfers (different pieces of our data mean different things), which would
have different processing rules, and the ability to push data from HQ to
the client sites.  So bidirectional data transfers are something we are
very interested in.  We also want to support multiple transport mechanism
(FTP, email, WebDav, etc) -- we should be able to send and receive messages
in many different ways.

Now, I am pretty sure that BizTalk2004 can support multiple ways of
receiving data, and multiple data formats.  So, for example, if we have two
types of messages, "UserData" and "SystemData", we should be able to
specify two different formats that the data must be in.  Correct?

I am a little confused how BizTalk2004 would actually implement our rules
for validating and importing the data into the main SQL Server database.
Do we write .NET code which is somehow processed by BizTalk?  Or do we use
some sort of internal BizTalk language?

I am a complete newbie to BizTalk if you could not tell. 

I had originally planned on implementing the solution as a Windows Service
which would monitor many receiving locations for data transfer jobs, which
could be of many types.  Each "type" of data transfer would have a routine
to import the data, validate the data, export the data, etc.

I had also planned on allowing users to configure jobs such that they could
specify what data is retrieved, manipulate it some how, and then package it
for export -- similar to SQL Server DTS.

So, basically I want to be able to do this:

1) Define processes for validating and importing different types of data
messages.  Some messages may require a human to "approve" the import.

2) Define processes for aggregating, transforming, and exporting different
types of data messages.  Some messages may require a human to "approve" the
export.

3) Be able to create new processes and have the mdistributed to each site
easily, preferably by the data transfer process itself.

I want to be able to use .NET code (specifically VB.NET) to write the code
to aggregate the data, transform it, export/import the data, etc.  But I
think BizTalk could be used to orchestrate the entire processes, right?

I apologize for the confused nature of this email, my thoughts are pretty
random at this point.

Any advice is greatly appreciated.

--
Mike Loll

[ Post a follow-up to this message ]

Hi Mike,

It sounds like BizTalk would be useful in implementing this scenario.

BizTalk provides an FTP adapter for connectig to the FTP servers.
BizTalk provices a SQL adapter for the database access.
Other adapters (SMTP) are available for other communications.
BizTalk schemas can be used for validating the messages, further validation
can be performed in orchestrations.
Human Workflow Services can be used where human interaction is required.
Orchestrations and maps can be used for the aggregation, and transformation,
(maybe you won't have to write much VB code to support this.)
Busness rules will be useful for hemping to define the message processing.

Having said all that, it sounds like you the project is complex, and it's
not easy to learn a product like BizTalk whilst you are in development.

It may be a good idea to install BizTalk, run through a couple of the
tutorials, then try building a Proof of Concept for a stage of the project.
This will give you an idea of that the product is about, and how will it
suits your project, without commiting to using it.

Regards,

Alan




"Mike Loll" wrote:

> Hello folks, I have a question regarding BizTalk 2004.  I have reviewed
> most of the Microsoft website's PR material, but I'm having a hard time
> making heads or tails of it, and BPI in general.
>
> If I could, let me describe some scenarios for you.  If you could, could
> you tell me if you think BizTalk 2004 would be a good solution for this or
> not.  Thanks in advance.
>
> My employer has an application which is deployed in most of the US states.
> It is used to collect data for our client, which is stored in a SQL Server
> database at each site.  Throughout the day, a task is run which exports
> data from the local SQL Server (based upon specific criteria) and sends it
> to a central repository at our client's main headquarters.  The data is
> sent via FTP.
>
> When HQ receives the data, it performs some processing, validates the data
,
> etc, and ultimately loads it into a SQL Server database.  An
> acknowledgement file is sent to the client site (actually it is placed in
> the FTP directory and the client downloads it).
>
> We are about to being a project to create significant enhancements to the
> uploading service, including porting the code to the .NET framework.
> Additional enhancements include supporting different "types" of data
> transfers (different pieces of our data mean different things), which woul
d
> have different processing rules, and the ability to push data from HQ to
> the client sites.  So bidirectional data transfers are something we are
> very interested in.  We also want to support multiple transport mechanism
> (FTP, email, WebDav, etc) -- we should be able to send and receive message
s
> in many different ways.
>
> Now, I am pretty sure that BizTalk2004 can support multiple ways of
> receiving data, and multiple data formats.  So, for example, if we have tw
o
> types of messages, "UserData" and "SystemData", we should be able to
> specify two different formats that the data must be in.  Correct?
>
> I am a little confused how BizTalk2004 would actually implement our rules
> for validating and importing the data into the main SQL Server database.
> Do we write .NET code which is somehow processed by BizTalk?  Or do we use
> some sort of internal BizTalk language?
>
> I am a complete newbie to BizTalk if you could not tell. 
>
> I had originally planned on implementing the solution as a Windows Service
> which would monitor many receiving locations for data transfer jobs, which
> could be of many types.  Each "type" of data transfer would have a routine
> to import the data, validate the data, export the data, etc.
>
> I had also planned on allowing users to configure jobs such that they coul
d
> specify what data is retrieved, manipulate it some how, and then package i
t
> for export -- similar to SQL Server DTS.
>
> So, basically I want to be able to do this:
>
> 1) Define processes for validating and importing different types of data
> messages.  Some messages may require a human to "approve" the import.
>
> 2) Define processes for aggregating, transforming, and exporting different
> types of data messages.  Some messages may require a human to "approve" th
e
> export.
>
> 3) Be able to create new processes and have the mdistributed to each site
> easily, preferably by the data transfer process itself.
>
> I want to be able to use .NET code (specifically VB.NET) to write the code
> to aggregate the data, transform it, export/import the data, etc.  But I
> think BizTalk could be used to orchestrate the entire processes, right?
>
> I apologize for the confused nature of this email, my thoughts are pretty
> random at this point.
>
> Any advice is greatly appreciated.
>
> --
> Mike Loll
>

[ Post a follow-up to this message ]

Mike - I agree with Alan. When you're considering a project as
'invasive' as yours, it would pay to spend a little bit of time playing
with the Biztalk samples, and a lot of time really understanding the
scope of the project, as the introduction of a product like BizTalk can
initiate major changes in internal business processes.

I have some fairly painful experience of clients who buy into BizTalk as
"the solution", and then try to retro-fit their existing businesses
processes around it, rather than spending the time working out the scope
of the project, and the appropriate use of Biztalk.

As Alan says, Biztalk has a number of tools within it that would suit
your project - adapters, pipelines, schemas, maps, orchestrations etc.,
but how you use them is more important.

I'd suggest reading "Enterprise Service Bus"
(www.oreilly.com/catalog/esb/) and "Enterprise Integration Patterns"
(www.eaipatterns.com) first to get a good understanding of the current
integration 'landscape' without the Microsoft/BizTalk spin, then working
out how best to implement a tool such as BizTalk.

Apologies if you know all this already,

Hugo

PS It always pays to ask an experienced consultant to help with this
sort of scoping exercise (cheap plug :-)).

[ Post a follow-up to this message ]

Mike

I believe BizTalk would be an excellent choice as part of the infrastructure
for your application. If it was me designing the application, I would not
hesitate
to use BTS2004.

When a message arrives at a BizTalk 2004 receive location, ( a File Receive
Location,
HTTP receive location, FTP receive location, Sql Server receive location etc
) the message is then submitted to BizTalk.
Processes in BizTalk then subscribe to the message, looking for a unique
namespace/ root node
combination in the message. This process in BizTalk (an orchestration) then
gets the message and
starts to process it. It is important to note that more than one
Orchestration can
subscribe to the same message. BizTalk out of the box can send and receive
XML/Flat File and
EDI documents. It can also send and receive other types of messages, such as
binary messages and
process them.

For the Human Workflow aspect, (human interaction with messages), there is a
built in framework called HWS (Human Workflow Services) that ships with
BTS2004.
Or you could build your own processes to email messages to person(s) for
approval (within a BTS Orchestration),
then these person(s) could send the messages back to BizTalk to continue the
process.

Another thing you may want to look at for Human interaction (GUI portion) is
Windows Sharepoint Services
and Microsoft Sharepoint Portal Server 2003.

BizTalk has an excellent mechanism to send out messages, and then wait for
response
back from an outside process. When the correct message comes back, it will
hook up
the correct message with the correct orchestration. BizTalk uses correlation
to accomplish this.

The mapper in BizTalk could help with your data transformations, for example
an incoming
PO is received and then an invoice and shipping document are created from
the incoming PO message.

For applying business rules to your processes, validating your data, BTS2004
now includes
the  Business Rules Engine. This is a great addition that was not present in
previous versions
of BTS.
Messages can be sent into the business rules engine for processing/validatio
n.
Once in the Business rules engine,  the message can validate your message,
change the message and then send the message back.
to BizTalk. The Business Rules engine can draw on facts to execute business
rules.
Facts include :
XML instances, .Net Assemblies and Sql Server.
The great thing about the rules engine, is you can change business rules,
and have the
new Business rules take affect without compiling and redistributing any
code.

I want to stop rambling about the Business rule engine but listen to this
webcast for more information :
http://msevents.microsoft.com/CUI/E...e=en-U
S


From within an Orchestration you can also easily call your own .Net
components, let
your code process the messages passed to it, and then optionally receive
back the
message or other message(s) for more processing in an orchestration. You can
also easily call Web Services
from BizTalk, or expose a whole orchestration as a Web Service.

From what you have described, if you decided to build your own BizTalk
engine, I think
you are going to be spending alot of time building/ modifying and maintainin
g
these custom processes. I would highly recommend to save yourself a huge
amount of time and effort
and use BTS2004 so you can concentrate on your business processes.

Matt


"Mike Loll" wrote:

> Hello folks, I have a question regarding BizTalk 2004.  I have reviewed
> most of the Microsoft website's PR material, but I'm having a hard time
> making heads or tails of it, and BPI in general.
>
> If I could, let me describe some scenarios for you.  If you could, could
> you tell me if you think BizTalk 2004 would be a good solution for this or
> not.  Thanks in advance.
>
> My employer has an application which is deployed in most of the US states.
> It is used to collect data for our client, which is stored in a SQL Server
> database at each site.  Throughout the day, a task is run which exports
> data from the local SQL Server (based upon specific criteria) and sends it
> to a central repository at our client's main headquarters.  The data is
> sent via FTP.
>
> When HQ receives the data, it performs some processing, validates the data
,
> etc, and ultimately loads it into a SQL Server database.  An
> acknowledgement file is sent to the client site (actually it is placed in
> the FTP directory and the client downloads it).
>
> We are about to being a project to create significant enhancements to the
> uploading service, including porting the code to the .NET framework.
> Additional enhancements include supporting different "types" of data
> transfers (different pieces of our data mean different things), which woul
d
> have different processing rules, and the ability to push data from HQ to
> the client sites.  So bidirectional data transfers are something we are
> very interested in.  We also want to support multiple transport mechanism
> (FTP, email, WebDav, etc) -- we should be able to send and receive message
s
> in many different ways.
>
> Now, I am pretty sure that BizTalk2004 can support multiple ways of
> receiving data, and multiple data formats.  So, for example, if we have tw
o
> types of messages, "UserData" and "SystemData", we should be able to
> specify two different formats that the data must be in.  Correct?
>
> I am a little confused how BizTalk2004 would actually implement our rules
> for validating and importing the data into the main SQL Server database.
> Do we write .NET code which is somehow processed by BizTalk?  Or do we use
> some sort of internal BizTalk language?
>
> I am a complete newbie to BizTalk if you could not tell. 
>
> I had originally planned on implementing the solution as a Windows Service
> which would monitor many receiving locations for data transfer jobs, which
> could be of many types.  Each "type" of data transfer would have a routine
> to import the data, validate the data, export the data, etc.
>
> I had also planned on allowing users to configure jobs such that they coul
d
> specify what data is retrieved, manipulate it some how, and then package i
t
> for export -- similar to SQL Server DTS.
>
> So, basically I want to be able to do this:
>
> 1) Define processes for validating and importing different types of data
> messages.  Some messages may require a human to "approve" the import.
>
> 2) Define processes for aggregating, transforming, and exporting different
> types of data messages.  Some messages may require a human to "approve" th
e
> export.
>
> 3) Be able to create new processes and have the mdistributed to each site
> easily, preferably by the data transfer process itself.
>
> I want to be able to use .NET code (specifically VB.NET) to write the code
> to aggregate the data, transform it, export/import the data, etc.  But I
> think BizTalk could be used to orchestrate the entire processes, right?
>
> I apologize for the confused nature of this email, my thoughts are pretty
> random at this point.
>
> Any advice is greatly appreciated.
>
> --
> Mike Loll
>

[ Post a follow-up to this message ]