Go Flow

I will be speaking at SOA World Conference & Expo, June 23-24, in New York City on the subject of “Making Sense of the BPM & SOA Standards Alphabet Soup“. 

Here is the description: What is BPEL? What is XPDL? How are they different? What is the best use for each? What is BPMN, and why should I care? Which of these are primarily designed to help IT at a technical level, and which help an organization at a business level? Amongst the flurry of BPM and Web services standards that appear to overlap at times, there is a central core of important ones.

I will also be speaking briefly about process discovery and another technology panel.

I just love that term: “Process Confabulation“. It sounds like something that WC Fields or Mark Twain might say. I saw it used in a slide share presentation from Michael zur Muehlen. What does it mean? It refers to an interesting problem in uncovering the process that a business organization is currently doing. Before any BPM project, you must first answer the question: “What is the current business process?” The first place you will look is in the organizations policy manuals. If such manuals exist, they are very likely to be out of date. Next you interview the people involved.  The catch is that much of their knowledge about the way they work is tacit knowledge: things they know but they can’t really explain to others. Yet if you ask a professional how they do their work, they can’t say “I don’t know”.  How do they justify their salary if they don’t know what they do?  They simply can not admit this.

So what do they do?  They confabulate: they make something up. They think carefully about what they think they should be doing. They then describe the business process which is somewhat idealized. It may be the processes that they think they do when they get the chance to do things right.  They may not actually be aware that they are making it up; it is a trick that the mind plays on us.  Because they are performing the job, it is inconceivable that they don’t know how to do the job, and as a result they believe whatever explanation they come up with.

Fujitsu has pioneered a new process discovery service to help avoid Process Confabulation. This process discovery service works by extracting data from log files that you already have in your organization. (See EBizQ, TMCnet, onSTRATEGIES.)  Think that you don’t have any such log files? Think again.  Almost every step of an important business process involves some sort of record keeping.  Currently those separate applications are not linked together in any way, but it is possible to take dumps of these database tables, find correlation values between them, and then produce diagrams of the business process.  As a service, Fujitsu can discover these processes in about a week or two.  The interesting thing about the diagrams produced is that they a factual representations about what has happened in the past. They are not confabulations. They serve as excellent reminders of the current process, which then those within the process can use as a starting point for talking about the process with those people who participate in the process. The key to the Fujitsu approach is that there is no need to put a sensor into your environment ahead of time: you can start immediately with the data that you already have.

Vance McCarthy from IDN pointed out to me this week that this new capability may have an interesting dynamic in the way that BPM is implemented. BPM project planning is done today with IT and business people together, but it is the business people in the driver’s seat defining the process, and the IT people simply implementing. Process discovery from data log files allows the IT people to show the business people the actual existing process. The discovered processes are credible, shifting the roles that the IT folks can play. It is hard for me to guess what effect this shift might have, but it does remind me of a trend that was discussed by Shoshanna Zuboff in her book “In the Age of the Smart Machine” about the transformation that occurred when retail grocery stores got barcode scanners; they were able to use the additional information about sales patterns in order to dictate to suppliers what they wanted, instead of the other way around. It is possible that process discovery from log data may be an empowerment to anyone who masters the technique quickly.

There is intense interest in automated process discovery.  These days I am getting so many queries that I may devote a few more posts in the future about some of the less obvious aspects of this developing field. I will try to use the term “Process Confabulation” as well — simply because it just sounds so unusual.

I am watching a number of comments being placed about a new effort for BPMN 2.0. Vishal Saxena says that the BPMN 2.0 metamodel should maintain this flexibility that BPMN 1.0/1.1 has. No argument there. Sebastian Stein says that BPMN is missing an exchange format, and clearly he does not know about XPDL. He goes on to say that the real problem is a lack of clear execution semantics. He points out that the OMG discusses two approaches: BPMN defines the semantics, and BPDM defines the semantics. Bruce Silver comments that the first approach would be the most value to the BPM community. We seem to agree that BPMN needs more clarity in expression. I suggest that there is a third approach that the OMG should consider.

My concern is that the BPMN committee is heading down a blind alley: One part design-by-committee, and two parts not-invented-here. It might be expressed as something like this: “Products can’t completely agree on the semantics of the drawings, so let’s just make up some new semantics, and everyone can just use that.” As if the problem was that products simply can’t define precisely what they mean, so the standards committee needs to define it for them.

The issue is NOT that existing products have undefined semantics! There are many products on the market, every one with precisely defined semantics of what they want to, and need to, express. The fact that products have inconsistent implementations of BPMN is not because they don’t know what to say. Instead it is because BPMN is not good enough at expressing precisely what that product needs to say.

Let me give you a concise example. In BPMN you can define an activity to have multiple outgoing arrows. Does this mean that all, some, or one arrow will be activated when the activity is completed? There are three cases to consider:

If you leave the tails of the arrows unembellished, it indicates AND split semantics and all arrows will be taken. If you put diamonds on the tails of the arrow, it means Inclusive Or semantics: one or both of the arrows will be taken dependent upon conditions within the diagram logic. The other normal case would be “Exclusive-OR Split” semantics, which means exactly one of the arrows will be taken. There is NO WAY to indicate in BPMN that the outbound arrows from an activity have exclusive-OR split semantics. It is simply missing from the standard.

This is surprising because so many products out there have traditionally made use of exclusive-OR. Traditional “flow chart” modeling uses exclusive-OR predominantly. Of course, there are other ways to draw it: I can make a single arrow go to a gateway, and there is an exclusive-OR gateway, so it is a logically equivalent way to draw it, but it is different drawing, and significantly more cluttered one at that. Not every product needs exclusive-or in this case, but some do, and we all would benefit if there was a standard way to express this.

Products are inventing extensions to BPMN because BPMN does not have the power to express precisely what they need to express. BPMN is the best standard notation we have, but it is not finished, and there are still many limitations. Now that we know there is a “meaning” to be expressed, and no way to express it, the BPMN committee should focus on extending BPMN with a way to express this. Don’t waste time developing a visionary model of a new non-existent product. Instead, the committee should be looking at existing products, and observing how they have used BPMN inconsistently, in order to figure out what gaps exist in BPMN today. Then come up with a clear expression so that when we see the diagram, we know what it means.

Many of the successful products today are based on years of research and customer trials which have directed the products to become very very effective at some particular segment of the market. How naive it is to believe that the difference between products are accidental and inconsequential differences. How arrogant it is to think that any group of experts can simply invent a new approach which is universally better than one that has already been proven successful. But this may be precisely what the committee tries to do.

The purpose of a notation is to clearly and unambiguously express something about the world as it is. It is NOT the purpose of a notation to redefine the world as we would like it to be, regardless of how attractive that idea might be.

If we think for a moment about the “musical notation” analogy (my previous entry) it would be a bit like the committee saying: “now that we have defined the musical notation, let us define instruments that can play such notes. We have a staff that goes from E to G, so I need an instrument that goes exactly from E to G”. The reason this is so laughable is that the sound of a musical instrument is infinitely richer than what is expressed in the notation. Reading the score of Beethoven’s 9th would be meaningless without having a familiarity with the different sounds of the instruments that are playing the various parts. The notation does not define the sound that is made, but it does express exactly what the musician needs to know in order to play along.

The best way to arrive at a more precise definition of the execution semantics of BPMN, is to look at existing successful products today, that already have precisely defined semantics. Examine how well these products are able to use BPMN, and particularly at how BPMN falls short of being able to precisely express what such a product needs. Then extend BPMN so that each product can clearly, and unambiguously, express what it needs to say. In this way, the BPMN standard would gain from all the best ideas which have been developed over the past 20 years.