SOA - Transaction/compensation/cordination  
Author Message
Siteadm





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

I have a challange.

I have multiple services that are located on different server setups. They are build stateless N-tier architectures based on single call remoting.

The services do not support the flow of transactions. This would also couple the services an go against the very idea of service orientation.

But ..nevertheless, one needs to perform business operations that span multiple services.

How do I best support this. Should I have the programmers do custom compensation code very time (clean-up) and accept that if something crashes the data is inconsistant or should I try to find/implement some kind of common cordination

What are my choices



Architecture4  
 
 
Roger_Wolter_MS





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

If you have related services that need to be atomic, your choices are transactions or compensation. As you said, distributed transactions pretty much eliminate loose coupling. Reliable, trnasactional messaging can dramatically increase the probablility of the services succeeding but it's still possible the all the services will have to bereversed and because rollback isn't really an option in a loosely coupled system, compensation is your best option. I have some ideas on the topic here: http://blogs.msdn.com/rogerwolterblog/archive/2006/05/24/606184.aspx and here: http://msdn2.microsoft.com/en-us/library/aa964144.aspx
 
 
Udi Dahan The Software Simplist





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

The fact that you have applications "located on different server setups" doesn't automatically make them services. You are right in staying away from atomic/2pc transactions that cross service boundaries.

If you find that you need to do quite a lot of compensating transactions beween services, I'd be open to the possibility of a less-than-optimal service decomposition. The loose coupling between services is first-and-foremost on the business level. Look for things that have a human involved in the workflow - that often delineates a service boundary, as does a long time-interval between operations.

My advice is to take a step back, rather than digging in and solving the technical problem.



 
 
Siteadm





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

Hi Roger

Thank you for answering my post.

Do know where I can find an example of an implementation of transactional messaging

I'm guessing that you are saying that I should look into using some queuing and maybe also some kind of work flow engine

I would like the solution to be as simple as possible because we need the functionallity very soon.

Sitedm


 
 
Arnon Rotem Gal Oz





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

I agree with Udi,

if you find that you need the transactional behaviour alot then the business logic of the services is coupled even if you don't technologically couple the services by introducing distributed transactions.

Arnon



 
 
Roger_Wolter_MS





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

There are several alternative for transactional messaging including MSMQ and >Net 3.0. I tend to like the Service Broker because of the added reliability and performance of doing the queuing in a database. Here are a few Service Broker references. Let me know if you need more:

http://msdn2.microsoft.com/en-us/library/ms345113

http://blogs.msdn.com/remusrusanu/

http://blogs.msdn.com/rogerwolterblog/default.aspx

http://www.sqlservicebroker.com/

http://www.amazon.com/Rational-Server-Service-Broker-Guides/dp/1932577270/sr=8-9/qid=1171647840/ref=sr_1_9/002-7616293-6809644 ie=UTF8&s=books


 
 
Ahmed Nagy





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

Hi Siteadm;

Can you give us more hints about your application. As the Udi and Arnon say, you do not to have to go for complex transactional system and implement distributed transactional system or Queued components. I would suggest few things to evaluate before making your decision...

  • Is your application one logical entity that spans its operations to multiple servers Or is it a trure Service Oriented Application
    • In case it is SOA you will need complex transactional system that can simply be provided by WS-AT in Windows Communication Foundation.
    • In case that your application has code that executes on multiple servers, you might consider COM+ which provide you with couple of advantages...
      • You can implement Remote transactions using COM+, this is fast and reliable
      • You might need to implement Queued components that provide you with 100% guaranteed message delivery.
      • COM+ will allow you to implement compensation for operations that cannot be rolled back automatically like writting to file system
  • Evaluate the physical communication reliability between your servers.
    • In case all your servers are in the same LAN, you go for transactional system using COM+ or WCF
    • In case you do not have reliable connections, use compensation techniques.
  • Does all the components that are located on different servers work on the Same Database or not

I'm sure you have other things to evaluate like your current system and existing code. Be careful about SOA as the term is sometimes misleading. If your application needs to expose functionality to other systems from a business prespective, then you have a true SOA.

I hope that was useful,



 
 
Siteadm





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

 

Hi Ahmed

Thank you for answering my post.

I have multiple applications that exponse multiple services. Each "application" is located on a seperate setup and is build on an N-tier architecture. The applications work with single sign-on.
They are all at the same hosting company so they can run LAN. The applications all have web service APIs that our customers can use. But my problem is now they all have to talk together

Many different front ends are to use the services. The front ends are not build for one "application" they might span more (without the user knowing it)

The services are rather fine grained. There is one service per business entity. Like the ERP application has services like Employee Service, Department Service.

Each application has an architecture like this:
http://tkfiles.storage.msn.com/x1pppQPzd7N2QxBQ4ZJX1ILBxCWWcDpzdXorj-NvVNX5ganwoWp-REQ4z8ccE-WHCEfRVpIpdZLOirY3hsSS8EgvYclPvY_5gkNyzeZzGQIHsA

I need to write to two or more applications at the same time and make sure that things get roled back if anyting goes wrong.

This is an illustration I used in another post.

http://tkfiles.storage.msn.com/x1ppUPyqopddk6CQ9nYZrZQklzqYvUYuorlmT4PiIXtzw9KXSibmc_jky8MUhCkTxS1vSLi3QKP6-5h9y-khgrS1cijSXzCU6m2NtfFV2fCV11A7awiRybxkaP3m9hxOtxA52dzygqM50tTTSwxR6hr_w
(Notice that I do not use Biztalk or a workflow based system - I made the diagram to try to understand someone). You might see the post:
http://forums.microsoft.com/MSDN/ShowPost.aspx PostID=1234104&SiteID=1. It is very related.

Notice that the architectures are stateless - they are buildt based on single call remoting.

I could really use any input that you might be able to give me.

 


 
 
Siteadm





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

Thank you Roger

Thank you for the great links. I'm registering for a virtual lab tonight so that I can check it out. I know nothing about Service Broker :(

I will mark your posts as answers. (and also the others with relevant input). Im just afraid to mark them as answers yet - and then not getting anymore feedback. I'm not sure I have found the right solution for my case yet.


 
 
Ahmed Nagy





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

Hi Siteadm;

According to MSDN, your current technology does not fit your current requirement. Read From MSDN: "It is important to note, however, that neither the ASP.NET Web services nor the .NET Remoting plumbing supports propagating a declarative transaction, so it is impossible for either sort of endpoint to inherit a declarative transaction via a cross-process call.". 

Reliability of connectivity between systems should make thinking about compensation comes in a second place provided that your transactions are short, optimized and independent from user input.

Now, I can suggest that you use WCF. WCF Supports many communication protocols and some models for distributed transactions including WS-AT (Web services atomic transactions), which fits your requirements. I'm glad to see the service and proxy layers in your architecture which means that migrating to WCF should have mimimum impact. You can find many resources on that including MSDN as a primary source of samples, see this link. Note that WS-AT requires special handling in terms of configuration.

I have a recent article about WS-AT and distributed transactions located here and it might give you some insight on how distributed transactions work. The article is conceptual and does not introduce WCF in depth, but it explains how WS-AT works.

I hope you find that useful,



 
 
Udi Dahan The Software Simplist





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

"I have multiple applications that expose multiple services"

That's another unfortunate overloading of the term Service. Same with Web Service.

When talking about services in the context of SOA, then a service is not exposed by an application. If anything, the service IS the application. Other software interacts with the service by sending it messages - calling a "webmethod" on the service (with or without parameters) is just another way to send it a message. The webmethod is not the service.

On the topic of stateless-ness, just because you don't keep objects in memory between webmethod invocations, that doesn't make a service stateless - state is still maintained in the DB. The fact that several webmethods need to be called in a specific order, passing the state received from one to the invocation of another, in order to get any business functionality done points to a problem with the granularity of the messages your services process.

For more information on entity aggregation in the context of SOA, take a look at my article "Autonomous Services & Enterprise Entity Aggregation" from the Microsoft Architecture Journal, issue 8.



 
 
Siteadm





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

Hi Ahmed

I had first thought of using WCF and WS-AT. The problem is that it would take require locks in the databases that might be envolved.

I'm quite sure that this would give me all kinds of deadlocks - since each "applications" services are also exposed to external developers.

As I see it I have the same problem as if the services belonged to different organizations. There is no way they would let anyone flow a transaction into their systems.

I think I need to find the best way to do compensation instead of atomic transactions. Right


 
 
Ollie Riches





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

Hi Siteadm,

As you say in one of your previous posts the services are 'fine grained' from the business process you are trying to model.

It sounds to me as though you do indeed need to use a compensation approach to your problem and this would be hosted and controlled from a higher level service ('Super Service' - awful name :)) that is utilising your 'fine grained' services you describe in the url's provided in your previous post - create another layer of abstract on top of the already existing services.

Obivously to support compensation in any of your 'fined grained' services will require funcrtionality to negate an operation supported, what I mean by this is if you had an operation to 'CreateOrder' you should also have functionality to 'DeleteOrder' for the same set of parameters - thus allowing you to 'rollback' any data you might have created\modified.

Now from my experience this all sounds quite simple in the abstract but it can get very complicated very quickly especially if you don't control the development or API's of the 'fined grained' services you wish to use compensation on.

HTH

Ollie Riches



 
 
Ahmed Nagy





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

Hi Siteadm;

Services exposed by the service layer should not be fine grained but rather should be coarse grained. The reason behind this is to minimize the number of possible calls cross application boundaries. So, at this point all your you can make your services stateless and always one call is enough to do the job. Always remember that cross process calls are expensive and you should always aim to minimize the round-trips between servers.

Back to your point about compensation, my advice is to use transactions where possible when all calls are under the control of your applications and no calls to external application is done. Even more, if an external application is calling ServiceA.ProcessRequest and this call triggers calls to other applications in your domain, you can still use transactions to start at the point where your application was first called. Avoid using transactions with systems external to your reliable network boundries (even if they support it). I do not see any reason that does not allow you to mix both techniques(compensation and transaction) where each one is applicable.

Compensation adds complexity to your solution so try to avoid it as possible or minimize its use. Think about using compensation when the following is true...

  • Operation is lengthy and locking might involve performance problems.
  • You understand and can evaluate the impact of having inconsistent data accross the applications during the time where certain request is being processed
  • You are calling external application that is not reliable in terms of connectivity, responsivness or availability.

I hope you find this helpful,



 
 
Siteadm





PostPosted: Architecture General, SOA - Transaction/compensation/cordination Top

Hi Udi

Thank you for answering my post. I'm sorry it has taken a few days to reply to it. I had to find time to read our article before I could reply. The article is great by the way.

There is a good vocabulary in your article that can help me describe the services to you.

I like the distinction of Process, Activity and Entity service that you mention.

The services that the applications expose are often entity services, but there is nothing stopping me from making one service that exposed all the services . I’m actually able to do transactions on the much related business calls for services inside the applications. Like in the diagram from the article (link below). I could make a service that exposes all functionallty of the CRM system - Much like the order processing service in your diagram. I am able to do distributed-transactions over multiple entity/activity services. There are messages that is sent today that include "Messages" with multiple entity types included. So they all get updated/created or rolled back.

http://msdn2.microsoft.com/en-us/arcjournal/bb245672.jour8autonomous04l(en-us,msdn.10).gif

Where my problem lies is in communicating with two services (In my world two appliactions). A CRM system and a ERP system. - also others.

I need to find a way for front ends to do "business transactions" that might span multiple services.

As I have come to see it. I believe that I should look at the applications/services as if they were located in different organizations - who would never allow the flow of atomic transactions - and then "garrantee" roll back by some kind compensation - maybe with a work flow engine with compensating work flows or SQL Broker . ..now I just have to figur out how to do that

I would very much like any further comments you might have or link to articles that might push me in the right direction.

Thanks,

Siteadm