Taiichi Ohno is credited with the creation of the Toyota just-in-time production system, and his book “Toyota Production System: Beyond Large Scale Production”  is a surprisingly good read even today when many of these principles are considered well established.

My interest was in understanding how this philosophy applies to Agile/Lean Software Development.  I was pointed this way by Tom and Mary Poppendieck’s excellent book on the same subject, as well as an insightful paper from Fujitsu on the subject of Innovation in Software Development Process by Introducing Toyota Production System (pdf).   As I am reading I am surprised at how perfectly the advice from Mr. Ohno fits to the advice I would give to a software team for Lean/Agile software development

Mapping Concepts to Software

Before the advice applies, you need to understand how software maps to 20th century automobile manufacturing.  Ohno’s central theme is to eliminate waste.

One form of waste is the consuming of parts (which cost money) into incomplete products (which can not be sold yet).  The most wasteful form of mass productions would be one where you build 1000 car chassis, then you put 4000 wheels on them, then you put 1000 engines in them.  All the time you have this inventory of 1000 incomplete cars until they are finally finished.  He would prefer that there be only a single incomplete car at a time.  The reality is that it takes time to put the parts together, and so there is an assembly line with a number of cars being worked on at the same time, and the real goal is a very steady even stream of not only the finished cars, but also all the parts flowing into the process.  The total accumulation of incomplete cars, as well as the accumulation of incomplete parts, adds to the amount of waste, and should be eliminated to the degree possible.  If you have a significant amount of over production, then there is additional waste from the need to store the extra parts and handle them multiple times.

For software, the bytes that form the lines of code cost nothing.  You should not think of lines of code as being the parts of the system, and you should not think about the source code as being the output of the process.  This focus on the source code as the “product” is one of the biggest misconception in the software industry.  Reality is less tangible.  The “parts” that are consumed are simply the “person-hours” that are invested in discovering and developing a feature.  This includes time from not only programmers, but all the various roles on the team.   The output is best described as  “satisfied customers”.  While slightly inaccurate, this phase is meant to remind us that it is not enough to burn a CD with the program on it, but the final sale is for installed software which is successfully used to solve customer problems.  If you have done the job right, they will be “satisfied”.

Just as in a car factory, you do not want to build 1000 cars in parallel, similarly in software you do not want to create 1000 features in parallel.  As you start to build a car, the waste builds as you add more and more parts, but the waste disappears when the car is completed and ready for shipment.  The uncompleted cars are a liability on the organization, and building cars in parallel multiplies the amount of waste.  Similarly, once your start designing a feature, and start investing person-hours, the waste builds in direct proportion.  Finally, when the new feature is actually in a “customer-ready” shape, that waste disappears.  The started-but-uncompleted features are a liability.  Features designed but not coded is a waste of time.  Code that is written but not tested is a real liability because there is now a hidden amount of work that must be done to correct problems in the coding.  Completed features in the code, but can not be installed are still a waste.  Building a bunch of features in parallel multiplies the waste.

Careful readers will notice that the above discussion is slightly simplified.  Even a perfect factory would have cars in an incomplete state, and a perfect software team would have some feature in an incomplete state.  It is wrong to call all items in transit “waste”, rather it is waste only if it is more than the minimum necessary.  We want to minimize the number of incomplete features, not eliminate them entirely.

While you can envision a software development team as a factory that produces “features”, you need to be very careful.  When building 1000 cars/day, all of the various steps are highly repeatable.  The time for each step can be measured to a high precision, and the exact costs of parts can be predicted.  For software development no such predictability exists.  You start a software feature with no reliable estimate of the cost.  This does not change the goal of elimination of waste.  Even though you don’t know ahead of time the amount of waste, it still makes sense to runs thing in such a way as to minimize waste in whatever amount occurs.

Quotes from the Book

Understanding the mapping, we can now take quotes directly from the book (including page numbers), and translate them in to meaning for software development teams.

The loom stopped instantly if any one of the warp or weft threads broke.  Because a device that could distinguish between normal and abnormal conditions was build into the machine, defective products were not produced.  – p6

Automated testing and continuous testing are a central pillar of Agile development.  Ohno is talking here about looms that were perfected by Toyoda Sakichi from 1910 to 1926.  Amazing that the concept of continuous automated tests was in use a century ago.

Stopping the machine when there is trouble forces awareness on everyone. – p7

When a test fails, a software team should drop whatever it is doing and address the problem of the test.  This does force awareness on everyone.

…the workers themselves should push the stop button to halt production if any abnormality appears. – p7

Stopping a production line is expensive, but here we see that Ohno understand that the expense of an unfixed problem is far greater.  The biggest enemy of software quality is ignoring test failures.  Once I joined a team where the building of the product produced thousands of warnings from the compiler.  The developers felt that warnings were harmless, but they are not.  First thing I had them do was fix the code that caused the harmless warnings, so we could then see if any important warnings came out.  Allowing anyone to check in source with a broken test propagates those broken tests, and that causes a blindness to the real problems in the code.

Repeating why five times, like this, can help uncover the root of the problem and correct it.  -p17

So much of software is written to get around problems at other layers of software.  Code bases often seem like a huge elaborate house of cards.  Compensating code is distributed throughout to hold it together.  One should ask: “why is that routine written this way”, “why does that support routine return this result”, “why is this value needed”, etc.  Addressing the problem at the lowest level will eliminate the need to multiple copies of the compensating code, and usually reduce the potential for bugs.

We regard only work that is needed as real work, and define the rest as waste. … we must make only the amount needed. -p19

Here Ohno explains an important concept that overproduction is waste as well.  Well known in software methodology as YAGNI.  Don’t implement anything until you need it for an actual end-user use case.  Contrary to this, programmers often pride themselves on “complete” implementations of interior object classes, commonly adding a host of getters and setters which are not (yet) needed by anyone, but they anticipate will be needed at some time in the future.  This additional coding is precisely the kind of overproduction that Ohno is speaking of: not only does it waste the programmer’s time to implement it, but the excess lines of code increase the code review and maintenance burden in the way exactly analogous to the need to store the extra parts in a car factory.

I used to tell production workers one of my favorite stories about a boat rowed by eight men.  One rower might feel he is stronger than the next and row twice as hard.  This extra effort upsets the boat’s process and moves it off course. -p24

Lessons on teamwork are always useful, and this points out the danger of measuring anything intermediate to “customer satisfaction”.  The goal in rowing is to get the entire boat to the end line as quickly as possible, and a focus on anything else, such as individual force on the oar can distract from this goal.  In software, getting a workable feature to the final customer is the goal, and measuring anything else, such a number of lines produced, or bugs found at a particular phase, can distort the process in unacceptable directions.

The operating method of the Toyota production system is kanban.  This piece of paper carries information that be divided into three categories: (1) pickup information, (2) transfer information, and (3) production information. -p27

Kanban was a way of decentralizing the control of production.  The factory is oriented on pull-based production, and the kanban is a way to communicate the detailed need for production, in order to avoid over (or under) production.  The kanban might be considered to be analogous to storycards that are used to communicate details of a customer use case.  This is only the top level, because kanban is used to communicate within the factory within any producing/consuming relationship, and for the most part this is NOT done in a software team.  Sometimes a bug report can be used to allow one team to make a request for enhancement on another team, and to track the progress.  In this case the bug report is analogous to the kanban.  But caution should be used in the analogy: remember that software parts are unpredictable by nature.  The very regularity of car parts, and the known quantity and cost of the parts, makes it very easy to implement specific requests for production.  In software, however, every part is unique and custom made for the situation, and so a much richer, more interactive discussion is needed.  Kanban should not be implemented literally in a software development team to replace internal communications.  Still, the idea that performance level is tuned according to the need to produce a given use case is very important.

I have good reason for emphasizing the role of top management in discussing the first rule of kanban.  …management commitment and strong support are essential to the successful application of this first rule. -p31

Here he is talking about moving control of production from the central planners, and replacing that with the more “autonomic” method of Kanban.  It is interesting to see how this must have been very uncomfortable for upper management to “trust” that the people on the floor would regulate things correctly.  It is the same with moving software development to an agile approach, where individuals on the team self-organize to get the current sprint done.  The waterfall approach gave more control to external authority, and also provided a better way to measure progress.  Changing to a lean software approach means that this control and to some extent this ability to report progress is eliminated.   This top management support is essential.

Unless one completely grasps this method of doing work so that things will flow, it is impossible to go right into the kanban system when the time comes. -p33

The key word here is “flow”.  In a factory the flow of physical parts is paramount.  In software, the flow of feature ideas is paramount.    The flow of continual builds, and regular sprints, with customer quality code available at all times is the key.  Stockpiling a huge number of features for implementation in an 8 month project is precisely like the mass production of cars that was beginning to show signs of inefficiency in the 60’s and 70’s.  Mass production creates waste.  Waste is eliminated by a steady flow of product releases, each quickly generated from the last.  This is the central theme of the book, and it is the central theme of Lean Software Development.

In the beginning, everyone resisted kanban because is seemed to contradict conventional wisdom.   … To make kanban understood throughout the company, we have to involve everyone. If the manager of the production department understood it while the workers did not, kanban would not have worked.  -p35

How interesting that Mr. Ohno had so much pushback.  He admits else where that it was non-intuitive, but remember that conventional wisdom is based exclusively on what you used to do.  We see the same with software method, including the need to include everyone.  If you train part of a team on the method, you will surely fail, (and I have direct personal experience with this).

While traditional planned mass-production system does not respond easily to change, the Toyota production system is very elastic and can take the difficult conditions imposed by diverse marked demands and digest them. – -p37

This is the promise of Lean/Agile software development: the continuous flow of small features allows for much more rapid response to market changes.  It is ironic that cars with all their physicality can be developed in an agile way, while software which due its lack of physicality aught to be easy to change, but is still developed with a waterfall model making it impervious to change.

Production Leveling:  It had been a long accepted production fact that continuous punching with one die in the press beings the cost down.  It was considered common sense to product in the largest lots possible and punch continuously without stopping the press. -p38

In the 1940’s, Toyota’s die changes took two to three hours … By the late 1960s it was down to a mere three minutes. -p39

The idea that large production runs can reduce cost is well established, and is exactly counter to the idea of producing only as many of a thing that you need.  Not deterred by this, Toyota figured out how to eliminate much of the waste in small production runs.  Ohno had faith that die changes could be made faster, and without that faith it would never have been accomplished.

The same thinking perpetuates waterfall software development: the established “fact” that there is overhead in checking in and testing code, therefor, if you have to touch the code you should implement as many features at once as possible.  If you have the orientation that testing is long and difficult, and you do it only once a quarter, then the idea of doing it once a day is simply impossible to consider, and quick development sprints equally impossible.  However, if you understand the advantage of continuous flow of features, then you automate your tests, and make them so you can run them every day.  If you have faith that tests can be automated and run quickly, then you have a chance of realizing this potential.

To ensure that we have 100% defect-free products, we must set up a syste that automatically informs us if any process generates defective products -p41

Automated testing, continuous builds, with automated runs of tests.  Require each programmer to run the tests before checking in, don’t check in if the test does not run.  This is Agile/Lean software development, and Ohno is telling us about it in the 70’s.

Rule three of kanban prohibits picking up or producing goods without a kanban.  Overproduction is automatically checked, even if someone wants to make more. -p43

If software is developed only to meed the needs of a specific use case, and the use case is attached to the check in, we will eliminate over coding of things not needed for the use case.

If one sticks to the idea that, once set, a plan should not be changed, a business cannot exist for long.   Sticking to a plan once it is set up is like putting the human body in a cast.  It is not healthy.   -p46

This puts a dagger in the heart of the idea that plans should be perfectly formed ahead of time, and this is particularly true in software where nothing is predictable.  Instead, a lean/agile approach to software allows plans to be flexible over time.  Very important, and an idea I frankly did not expect to come from a car manufacturing giant.

The plant should be a place where such judgments can be made by workers autonomously. -p45

Again a common theme in lean/agile software development approaches, but not what I expected to hear from a manufacturing giant.  In this case he describes the business organization as having an “Autonomic Nervous System”.

Similarly we want information only when we need it.  Too much information induces us to procese ahead, and can cause mix-up in sequence.  In business, excess information must be suppressed. -p50

The kanban allows the information to be carried with the products/parts, one of the reasons why kanban works.  It is interesting to see the bias against too much information.  He talks about automatic fine tuning in this quote:

as long as we cannot accurately predict the future, our actions should change to suit changing situations. -p52

Again we see flexibility and adaptibility as a core value, the antithesis of putting together a perfect plan and then executing long term against that plan.  Instead, adapt as you go along.

In any manufacturing situation, we frequently see people working ahead.  Instead of waiting, the worker works on the next job, so the waiting is hidden.  If this situation is repeated, inventory begins to accumulate at the end of the production line or between lines. -p59

Here we touch upon a very pernicious area of waste in software products.  If the schedule is relatively fixed, when a programmer finds that an implementation went well and they get done early, they often turn to implementing an extra pet project they have been intending to implement.  It is routine for development team to report to product management that few extra features have been included “for free”.  The worst cases of these I have seen have actually caused delays in the product because once implemented we found flaws in the approach, but it was too much work to pull out the functionality, so instead extra work went into fixing it.  Extra work is not extra value, it is instead increased waste.

Making large lots of a single part — that is, punching out a large quantity of parts without a die change — is a common sense production rule even today. … The Toyota system takes the reverse course.  Our production slogan is “small lot sized and quick setups.” -p95

In this chapter he compares Ford production system to Toyota system in order to emphasize the differences.  The parallel of the Ford system to waterfall software method is strong, while the Toyota system is like Lean/Agile software development.  The biggest single driver of continued use of waterfall is the idea that creating a bunch of features at once will reduce overall cost.  There is a feeling that there is an “overhead” to the cycle that can not be avoided, and therefor do as much in a single cycle is the most efficient way.  Toyota’s concept of flow and production leveling is just as radical as the idea of continuous builds and fast sprint feature cycles.  Both are are not intuitive, and have to be seen to believe that it is possible.  It should be obvious, however, that large lot sizes will make changing the product more difficult and maybe nearly impossible.  Thus it is the method of manufacturing that causes the inability to respond.

For automation to be effective, we must implement a system in which the machines sense the occurrence of an abnormality and stop themselves.  In other words, we must give the automated machines a human touch — enough intelligence to make them autonomated and achieve “worker saving” rather than “labor saving”. -p113

In software development, of course, machines are replaced with automated script and programming that does steps of the process, and these must include tests at every step.  Ohno’s statement here is strong: the automation without the sensing causes more waste than it is worth.  Thus tests built into the automated process, or possibly into the product itself, becomes most important in automating builds and steps in the software cycle.

Conclusions

In 117 pages he sets out a clear description of how Toyota has accomplished the climb to be the world’s biggest car maker, and is a must read for anyone interested in industrial production strategy.

Then, if you understand the mapping to software development, you find that he sets out an equally clear description for Lean/Agile software development.  If Ohno was alive today, I am convinced he would have been an eloquent proponent for developing software in short iterations, with continuous builds, continuous testing, immediate response to build breakage, and guided by feature burn-down.  In other words, a proponent of Agile Software Development.

You know that book on how everything important is learned in Kindergarten?  Along that same line, before I got into Kindergarten, my mother taught me to that if I put my toys away, I will be able to find them again later.  I am sure there was a lot of crying and whining involved, but like most people I eventually got the point.

Fast forward to the adult world.  How many times have you heard these questions:

• Where is the latest spreadsheet?
• Does this document have the latest changes in it?
• Are your changes in this document?
• Can you send the copy of the file that contains all the latest updates?

This situation is caused by the worst scourge of our time: the addiction to email.  Specifically the attempt to use email as a document management system.

The solution is so simple: find a place to put the document, where everyone can access it.  Put it there, and always update it there.  If you always get the document from there you will always have the latest copy.  If the always put updates there, everyone else will always get the change.  This is so obvious that I am sure most of you reading this will feel slightly insulted that I am spelling it out in such detail.

Why is it then so difficult to teach people this behavior?

Part of it is simply short sighted thinking:  it takes extra effort to put your toys away, and the benefit of putting them away might be a long time from now (like tomorrow).  Just like it is easier to leave your toys sitting in the middle of the room, when someone requests a document, it is simply easier to just email it to them.  Attach it as an attachment, and off it goes, problem solved … for the moment.  It takes quite a bit longer view to realize that (1) others may come back and request the document again, and (2) even that person is going to in the future want updated versions.  Why put out extra effort now, in order to save that work which will come in the future (like tomorrow).

Some of it is blindness to the dynamics: documents change over time, but for some reason we always act as this version I am sending now is the final version never to be modified again.  I laugh every time I see a document with a title that includes “FINAL” in the name.    It is even funnier when the title has multiple repetitions of FINAL.  How does this blind spot persist when at the same time we all have such difficulty locating the “updated” copy.

Some of it is selfishness:  I received an email this week from the finance department detailing exactly how to handle a particular kind of financial transaction.  From the detail of the procedure involved, it was obvious that the entire company was not expected to memorize the exact procedure.  They apparently expected everyone in the company to carefully file this important information away in a place that you can find it if the need ever arises.  Instead of taking the time to organize the information in a single place where everyone can get it, they push this task out to everyone in the company.  Multiplying the real effort to the company by many fold.  Clearly it would be better for everyone to integrate this into a corporate knowledge base, organized so I can find it if I needed, at that time that I need it.  But that kind of foresighted thinking is rare when it is just so easy to send an email.

First Step is Acknowledgment of the Problem

Let say that everyone realizes that email attachments are the problem, what can we use instead.  There are some problems with current document management systems (DMS).

Access :  I can send an email to whomever I want, and attach the document.  But putting a document into a DMS, how can I guarantee that the intended recipient can access the document?  Most internal corporate networks have become byzantine mazes as the result of attempt to control who can access what through the use of blocking certain addresses at certain routers.  I have heard this complaint from many people at many organizations.  The blocks are put into place in the name of safety, and getting thing unblocked is nearly impossible since it is virtually impossible to prove that such a change is still safe.  Desktop computer A is cut off from desktop computer B because a virus might propogate along that path.  How can you prove that virus will not?  You can’t.

Access Control:  If you are lucky enough to find a DMS that everyone can access, how do you  make sure that only the intended recipient(s) can access it?  All DMS have fine grained access control, but it is usually a large amount of trouble to change it because of the detail that you have to work at.  Addressing an email message is easy, especially if you use “Reply”.   Assigning users to access a document is easy, but there is no analog to “reply”.   Few DMS offer a way for people to request a document from those who own it.  Most DMS don’t even allow people to be aware of documents they don’t have access to, which would be necessary to make the request in the first place.

Transitive Access Control: If I send a document to Joe, he can forward it to Mary.  If I give right to Joe to access a document, most DMS systems will NOT allow Joe to give access to Mary.  It wouldn’t be “control” otherwise, would it?  But to allow a DMS to replace email, this is exactly what is needed.   In the Process Leaves system, we implemented such ability to “forward” the rights to another person, but most users find this so surprising and unexpected that they never use it.

The solution is to make a shared “room” where all the toys can be shared equally within a group.  That is the solution that many approaches have taken, and it is not difficult.  But someone still has to set up the room in advance, in anticipation of the need to share, and most people will not take this step.  It is just easier to send the documents as an attachment and force the work onto everyone else.  In groups that I work with, even making the room available to people, they rarely get used.

Any ideas on how to break the email addiction? Any tricks you know of in getting people to use “places” to put documents?  I made a similar plea last year in “Page First, Then E-Mail, Please” about the email message itself.  Otherwise, I am afraid, it is a bit like teaching teenagers the benefits of a tidy room — in other-words: a lost cause.

I could call this post “Removing the Risk from Lean Process Improvement” because it starts with the assumption that you want to improve your processes using Lean principles, but you want guidance on how to apply those principles most effectively.

Soooo much discussion of Lean last week at the Forrester Forum and the Gartner BPM Summit.  Who can argue against Lean?  It is after all a focus on providing more value with less waste.  Lean is a focus on eliminating waste, the original sevens wastes identified by Toyota, as well as elimination of anything that does not provide value to the customer.  We all want to get rid of waste and inefficiency.

How do you identify the waste in your business process? This is harder than you might think.  Sure, you probably have many anecdotes about cases that went terribly wrong, but how much did that really cost?  How common is it?  After all, one extreme case does not signify a trend.  What you need is a quantitative measure of how much your current process inefficiencies are costing you.

This is exactly what Electro Scientific Industries, Inc. (ESI) wanted to do.  They knew they had process inefficiencies for two reasons: they were having inventory problems and they were having a hard time responding quickly to customers.  But what was causing this?  ESI embarked on a Process Mining exercise using Fujitsu’s Automated Process Discovery.  What they found astounded them.  What they expected to be a 5 to 7 step process, with maybe a dozen variant paths, turned out to be far more complex in reality.  They analyzed 12,000 cases that had occurred in the past 3 years, and they found 1300 distinct process paths.  They found actual evidence for hundreds of cases of severe repetition, including one case where the order was changed 120 times!  They were able to identify errant paths, and tell precisely how many cases had gone that way.  They found specific process steps with unusually long average transition time, and could measure exactly how many cases were effected by this.  All of this information was pulled out of their existing SAP system without the need to install any new software in their environment.

This is exactly what you need for a Lean Process Improvement project. Greg Mueller who headed up the project at ESI, was able to show precise measures of the cost of process inefficiency to upper management.  This resulted in ample support for initiatives to fix them.  Peter Schooff did a podcast with Greg on exactly how they went about this.  As Greg says: “Process Discovery found 17 different ways to eliminated waste from their business process, any one of which would have more than made up for the effort of doing the discovery.”   Also, see the blog post from Sandy Kemsley covering this use case.

Being able to get precise measures of waste that can be eliminated is critical, even more amazing is how quickly this evidence can be gathered.  Some of the most important evidence was exposed on the second day of investigation.  The data gathering was completed in the first two days.  Analysis continued with what is best described as a “learning” phase: gaining a better and better understanding of what was really going on in the organization.  The team would slice the hypercube different ways, isolating teams from each other to discover process patterns in different offices or for different situations.  That data collection is still available today for immediate access so that when someone comes up with an innovative idea to eliminate waste, they can go back to the historical data and estimate the amount of savings that will generate.  The analysis never ends, but within two weeks they were able to identify 17 specific projects to eliminate waste from the business process, along with evidence based estimates of the savings to be made.

For those of you planning Lean Process Improvement projects, visualize sitting there, one week from today, with specific evidence based assessments of where your biggest process inefficiencies were, and how much they were costing you?  How much would that be worth?  Would that help you get funding and support for your project?  Sure it would.  You owe it to yourself to at least investigate what a process mining session might be able to do.

Fujitsu has been offering this service for about a year.  The biggest barrier is that is it just so hard to believe that the results are possible.  It seems too good to be true.  I admit, if I had not seen the results myself, I would be skeptical.  But I have seen the results (1, 2) which are nothing short of amazing.  Anyone considering process improvement will be wise to learn about them as well.  Informing yourself up-front with quantitative measures of the waste in your current processes, will eliminate much of the risk in a Lean Business Process Improvement program.

There might be three distinct kinds of process support necessary:

1) System Centric Processes
2) Human Centric Processes
3) Knowledge Worker Processes

System centric processes are fairly well defined: it is a kind of software engineering for very complex distributed systems. These are designed to completely automate the information flow so that no person has to be involved.

Human Centric Processes are those well defined processes which involve people. Since people are not “invoked” the way that software services are, there are a lot of extra capabilities that are needed in order to interface successfully with the asynchronous nature of personal interactions.

Are Knowledge Worker Processes distinct type of process, or just an extreme case? The unique thing about knowledge work is that it is unpredictable. Human Centric Processes can still be formal processes that are all predefined. Knowledge work, by it very nature, can not be predicted in advance. There is no fixed process for this kind of work. The work is “impromptu” in that the sequence of steps is figured out at the time that the work is done.

Some have called the technology to support knowledge workers “Case Management“. Especially in the medical field where you have experts who examine the “case” (the collection of documents and reports compiled for a given case) and then decide what to do next. (See CSMA.)   Case management is also the name used in the legal field.  See particularly work at National Center for State Courts which is promoting an idea of Configurable Case Management.   Some also call this area Dynamic BPM. Some call it “Unstructured BPM”. A better term might be Situational Process Management.

Where exactly is the boundary, if any, between Human Process and Knowledge Worker Process? Clearly KWP need extra support at run time to allow processes to proceed in ways that were not predicted in advance.

Thought Leader Summit Workshop

WfMC will be holding a Thought Leader Summit on this subject on Nov 3 in Maidenhead England. Please find additional information at:

http://wfmc.org/november-member-meeting.html
http://www.xpdl.org/nugen/p/lpqyjklyf/public.htm

The purpose of the workshop is to clarify the requirements of “the kind of process support which is needed in order to support knowledge workers”. What is and is not included in this kind of a product / technology?  This is the essence of this discussion hosted by WfMC. Henk de Man from Cordys will be there. Dana Khoyi from Global 360 will be there. John Hoogland from Pallas Athena will be there. Max Pucher from Isis Papyrus will be there. Many other thought leaders including Justin Brunt, Nathaniel Palmer, Keith Swenson (myself), Robert Shapiro, are expected to attend in order to get to the bottom of this highly relevant issue.

Goals: Determine the best name; create a short and long definition of the capabilities; get agreement on a list of requirements that such technology must have to be considered in the category; draw up a “Reference Model” or “Notional Architecture” for the category; follow through with a site for market education.  It is a tall order.  To those coming: be prepared to roll up your sleeves. 

The summit will continue on Nov 4 with a workshop on BPMN 2.0 portability. Will BPMN diagrams be exchangeable between tools? Will BPMN 1.2 diagram be portable to BPMN 2.0?  Answers to these questions will be worked on as a continuation of the fine work of the BPMN 2.0 / XPDL 2.2 working group.  This group has already completed a proposed BPMN2.0 serialization schema, prototype transform from XPDL2.1 to proposed BPMN2.0 serialization, and prototype transform from BPMN2.0 serialization to XPDL2.1.  This workshop will continue with discussions of BPMN 1.2 Status and use today, how fast people are migrating to 2.0, diagram interchange conformance classes, tools for validation, BPMN 2.0 Finalization Task Force Status, concerns over the direction, and other working group activities.

I collect nuggets of wisdom on various topics. Recently I have been going over the topic of Agile software development; what really matters?  Below is a list of 26 key principles to guide an agile software development team.

• Get case 1 fully working before starting case 2. Another way of saying this to use a kitchen metaphor is: “Serve the current meal before starting to cook the next“.  The biggest problem with software development is to start a bunch of things in parallel, because inevitably work will include something that is later discarded, meaning wasted work.  Work on one case (or a small number of cases depending upon team size); get that fully functional; get the tests running; write the documentation; check it all in as a finished piece of work, before you start on the next case.
• Never break the build.  Pretty obvious, but must be included in any list of software development advice.  A programmer who is taking all the proper precautions to test before checking in will never break the build.  If the build is broken, it is always because someone took a shortcut.
• Never implement a routine before it is needed in a use case.  When implementing a particular class, you should have a particular use case in mind, and you should implement only the methods required for that use case.  You might think about  the potential for other capabilities on a class, and you might document this in a comment, but implementation should wait until it is actually needed in a use case.
• Never add a data member before it is needed in a use case. Exactly like above except with regard to class data members.  It may seem obvious to the programmer that a “customer” record will need a “ship to address”, but that ship to address should not be implemented until you have a use case which specifically needs it.
• Don’t be afraid to make a decision;  don’t be afraid to change an earlier decision.  Agile development is about quickly responding to uncertainty.  Early in development you do not have complete information.  You should delay decisions as long as possible, but there comes a time that a decision is needed to move forward.  You can not hold up decision until that information comes in.  Instead, make the best decision you can on the available information.  Later, when new information arrives, don’t be afraid to change that decision.  (Some dinosaurs call this flip-flopping, but I call it reacting to a changing environment.)
• Continually learn how to improve quality. This job never ends, so you should expect to be constantly on the look out for things that could be improved, and collect examples of ways that quality problems were identified and addressed.
• Measure, measure, measure. Agile development helps address the problem of uncertainty about the future, but there should be no uncertainty about the past.  Tests should be continually running.  Performance of every run should be measured and recorded.
• Design around people, not systems. Too often developers get sidetracked into designing for technical opportunities. Never lose sight of the ultimate purpose of the software, and that is to help people get work done.
• Tests are part of the product.  Many developers and managers consider the product to be what you ship to the customer, and everything else less important.  The tests should be considered an actual part of the product, worthy of full consideration during design, and even, in many cases, delivered with the product to the customer. (This latter part is controversial, but a built-in self-test as part of a software delivery takes inconsequential space, and yet provide tremendous benefit when needed.)
• Write the test before the code. The test itself can be used to clarify the design for exactly what is needed.   Many times there are flaws in the design approach which are discovered when working through the test cases.  Think how much time would be saved to work through those cases before coding.  But: write the test for case1, and code for case 1, before starting case 2.
• Eliminate Waste. Frankly, another ubiquitous platitude which must be included in any list of development principles because it is so important.  There is no end to the job of looking for waste where it exists and eliminating it.  Eliminate anything that does not add value to the actual customer.  If you can not identify the value to the customer, then you probably don’t need it.
• Build a culture of immediate response to build breakage. Understand that when the build is broken, it effect everyone in the project, and so there is nothing more important than making sure that the central core code is building and testing properly.  I have seen teams that allowed broken tests to persist for months because is was someone else’s job.  Everyone suffered, but nobody acted.  Instead, there needs to be widespread recognition that a little work will pay back in a big way over the team.
• Every team members needs to understand the needs of the customer.  Large complex projects must be broken into separate teams and further divided for handing out to developers, but this should never be done to the extent that people lose sight of the desires and goals of the actual users of the final product.
• Keep related definitions together. Structure the code so that highly related things are located together, possibly within one class.   This is a standard OO design principle of encapsulation.  Ideally, all the code outside the class will not need to know the details of the internal workings.  Some developers delight in spreading details across multiple files in order to organize in different way: such as to keep all the same data types together, or to organize alphabetically.  For example, putting all the constants in one class in a different package from where they are being used adds unnecessarily to the complexity of the program.   The guiding rule should be to group by relatedness with the result being to hide complexity.
• Always run the tests before checking in. This guideline will help you satisfy the  “never break the build” guideline.
• Premature optimization is the root of all evil. A quote from Don Knuth which rings true today.  Code should be written well to avoid needless waste at the micro level, but optimization beyond the individual method level should wait until testing within the entire program with a stress test bases on an actual end user use case.  Intuition of what is important for overall performance is almost always wrong when based only on a static understanding of the code.  Instead, measure the behavior of the complete system, to identify the 1% of the code that really makes a different in performance, and focus on that.
• Minimize backlog of uncompleted coding tasks. When a developer starts to work on a use case, there is a cost associated with all the code that has been modified, but not completed and tested.  Holding uncompleted changes for days or weeks adds up to a significant risk of waste due to rework.  Consider three tasks estimated to take 1 day each.  Starting all three at one time, and working in parallel for three days involves an accumulation of 9 “units” of cost.  But doing each task sequentially, completing each task before starting the next, involves only 3 “units” of cost.  This is not intuitive.  Our intuition tells us that while we are in there, we might as well do three things at once, before buttoning the work up.  But software is not like physical construction.  Short, quick, and complete jobs not only cause less cognitive load, but also reduce the chance that uncompleted work will conflict with another person’s uncompleted work.
• Never overgeneralize functionality. This is also know as “YAGNI – You Aren’t Going to Need It” .  While coding a particular class, programmers like to think with a small tweak this class might be used for several other purposes.  This is fine if those purposes are required by the current use case, but usually the programmer is thinking about uses which have not been invented yet, and which may in fact never be needed.  (This subject always reminds me of the classic Saturday Night Live skit about the product which was both a floor wax, and a dessert topping.)
• Never use 3 lines when 2 lines would do. Succinctness in code pays every time someone else has to read it.  But don’t shrink the code to the point of being difficult to read.  Smaller, well written code can easier to maintain and easier to spot errors in, than verbose, ornately written code.  Always simplify as much as possible, but no more.
• Never ever measure code by counting lines. The number of lines needed to do a particular task varies greatly from programmer to programmer, and from style to style.  The number of lines of code does not tell you much of anything about the completeness or the quality of the code.  Code quality can vary by a factor of 200, and this overwhelms any usefulness of the count of the lines.  Count instead functioning use cases.
• Continually re-design and re-factor. Apply this cautiously because some code is brittle and difficult to change, but in general you should not be afraid to change the code to match the real use case.  A data member may have been an integer in the past, but when a use case requires it to be a floating point don’t be afraid to change it, only be sure to complete everything: tests, documentation, installer.
• Delete dead code. There is a tendency to let “sleeping dogs lie” when it comes to large blocks of code that is not well understood.  One example is adding a new method to a class to replace another, quite often the developer will leave the old method there “just in case”.  Some effort should be expended to check to see if that method is needed, and delete it if there is no evidence that it is needed.  The worst offense is commenting out blocks of code, and leaving that commented code around.  Commented code should be deleted as soon as you know that the tests run, and certainly before checking it in.  It is easy to add code at any time, it is hard to delete code at any time.  Therefor, at a particular time that you have a good idea that something might not be needed, and small extra effort to verify this and eliminate the code will make the codebase more maintainable.
• Don’t invent new languages. Programmers love to make text files that drive functionality in way configurable at run-time.  There are no end of configuration files to be able to change the behavior of the program without recompiling.  The advent of XML is driving a chain of specialized custom “scripting languages” that allow functionality to be “programmed” by the end user without having to compile.   The flaw in this reasoning is that the precise definition of the behavior of the operations almost never well defined outside of the context of a particular implementation, and these types of scripting languages are mainly useful only to people who have an intimate knowledge of the internal working of the code body in question.  Thus, real end users without detailed internal knowledge can never possibly know what is necessary to anticipate the effect of complex combinations of commands.   Scripting languages have a use, and can not be eliminated, but the designer must take a very very conservative approach and use existing languages as far as possible, and avoid inventing new ones.
• Do not create a design until you are ready to implement and test the implementation. You should have some overall idea of where you are going, and a overview of the system architecture that will be aimed for, but no detailed design, no detailed description of functional implementation should be written down until the development iteration that will allow that design to be implemented and tests.  The detailed design should cover only as much as is needed to handle the current use case.  The biggest cause of waste in software development is time spend designing things that are not needed or need to be redesigned because of some mistaken assumptions that the design is based on.
• Software is Plastic. Unlike physical manufacturing, software can be changed in significant ways very easily.   In fact there is plenty of evidence that software is easier to change than the design specifications that describe the software.  Furthermore, software communicates the design more effectively than the specification.  Therefor, you should spend the time to implement the design directly, so that customers can see the details of the design.  If you miss and have the change the design, it is easier to change the software than it would be to change the spec.   But most important, your information about what the customers wants is far better after they have seen the code running.
• Take the time to code a complete description of the problem in the code that detects and reports exceptional situations. Programmers are often very lazy and throw exceptions with superficial descriptions of what is wrong.  Thinking that they are the only people who will ever see the problem, and they will remember the meaning of the problem from the vague description included.  But in fact more time is wasted in customer support situations because of inaccurate or incomplete error reports than any other cause.   Write every error message is if you are explaining the situation to someone who just walked into the room and has no experience with the code.  The customer, and the customer support team, after all, have no experience with the code.

These are presented in no particular order.  I welcome comments on principles that I have left out, or (if this is the case) principles that you disagree with.

John Evdemon makes an informative post about the  Zachman Framework and end up on a discussion of  “process reification” which he explains is the action of converting an abstract process into a concrete, executable process.  Such a conversion is often touted as the Holy Grail of BPM.  While such a conversion is often desired, I would remind the reader that it might be better not to convert in the first place:  A Model Preserving Strategy assures that the designer creates an executable model in the first place, and that the model executed is always what the designer drew.

JP Morganthal makes an important post reminding us to Keep Your SOA and BPM Initiatives Separate.  Read through the comments and you will see precisely why there is a problem in the community.  I have met so many people that think that BPM is just a scripting language for SOA services.  These people will argue at length that “BPM is a part of SOA because SOA is useful without BPM, while BPM is useless without SOA” (this is an actual quote from a noted BPM expert who shall remain nameless for this post).  JP makes the point very well, but let me put my spin on it:

• SOA is ultimately about “code reuse” and simplifying code maintenance.  Instead of distributing a piece of code throughout your computing infrastructure, you can deploy it in one place and call it from anywhere.  The goal of SOA is optimization of IT resources.
• BPM, far from being simply a scripting language for the above, is the practice of supporting office work to accomplish business goals.  The goal of BPM is improving your business.

Yes, if the business process needs to retrieve information at a point in the process, then SOA is recommended but OOA (Object Orientated Architecture) will work as well if that is all you have.  I too have seen projects get derailed by SOA when a less modern approach would have supported the business process as well.  I think it was put best by the CIO of a major bank stood up in a session I attended and said “SOA is a tool we use, while BPM is what we do.”

So much discussion recently about Case Management, but do we really know what we mean?  Let me collect here some definitions, and then offer my own.

The Case Management Society of America, a health care oriented professional group, defines case management as:

“a collaborative process of assessment, planning, facilitation and advocacy for options and services to meet an individual’s health needs through communication and available resources to promote quality cost-effective outcomes.”

While the health care industry is a large user of Case Management, they are not the only user.  One legal site defined case management as:

techniques used to process cases from one stage of the proceeding to another, such as setting deadlines for discovery or scheduling a series of pretrial conferences. Case management calls for different approaches from one case to the next and is the primary responsibility of judges, assisted by lawyers and clerks’ office personnel.

In the legal profession we have the terms “case file” (a complete collection of documents for a case) and “case law” (the law as laid down in the decisions of the courts).  There is no question that a “case” is a unit of work; it is a package of some sort.  The case is used as a focal point to collect all the information about a the work.  There may be a goal, but somehow the case exists independent of a motivating goal.  It exists as something to be worked on, and thus case management is a way of organizing or accomplishing work around the case.  A mental health care site offered this:

Case management is the coordination of community services for mental health patients by allocating a professional to be responsible for the assessment of need and implementation of care plans.

Notice that decisions are made as part of case work.  There is a clear feeling that the work to be done is not precisely defined, but rather you have to figure out the work to be done as you go.  You have the information, but there is an assumption of judgment on the part of the people working on the case.

Law enforcement as well is familiar with cases which are used to represent current investigations, and case management is the handling of those investigations.

Other definitions devolve into specific expectations of what one migh expect to achieve using case management, and I list a few of those definitions here for context:

The linking of a consumer to the service system and coordinating the various system components in order to achieve a successful outcome. The five case management activities are: (1) assessment, (2) planning, (3) linking, (4) monitoring, and (5) advocacy. Case management’s primary goal is service provision for the consumer, not management of the system or its resources.

Coordination of services to help meet a patient’s healthcare needs, usually when the patient has a condition which requires multiple services from multiple providers.

An individualized plan for securing, coordinating, and monitoring the appropriate treatment interventions and ancillary services necessary to treat each offender successfully for optimal justice system outcomes.

Offers a single point of entry to the aging services network. Managers assess clients’ needs, create care plans, and coordinate and monitor services. They may operate privately or may be employed by social service agencies or public programs. Typically, case managers are nurses or social workers.

A method by which a health plan attempts to control costs by directing all of the procedures for care of an individual through a nurse or other health care professional.

While many of the above definitions attempt to explain what Case Management might accomplish, and what it might mean in a particular situation, we can certainly draw some commonalities out of them:  there is work to be done, but it hard to say ahead of time exactly what that work is; there is a collection of information (documents) which are needed by people in order to make intelligent decisions about the course of actions which unfold as the work progresses.

Can we come up with a generic description which is free of biases of a particular industry? How about:

Case management is a coordinative and goal-oriented process, to handle cases from opening to closure, interactively between an internal or external client and a case manager or case team.

BP Trends offers in July 2009 this definition:

Case Management is the management of long-lived collaborative processes that coordinate knowledge, content, correspondence and resources to progress a case to achieve a particular goal; where the path of execution cannot be predetermined in advance of execution; where human judgment is required to determine how the end goal can be achieved; and where the state of a case can be altered by external out-of-band events.

This is a strong attempt at providing a definition that includes many characteristics and is independent of any particular industry.  It is, however, a mouthful and too detailed to be used in everyday conversation.

A Modest Proposal

I am comfortable using the term “process” to describe a collection of activities, even if they are not strictly ordered.   Some purists might insist that a process must be pre-defined and strictly ordered, but I would refer them to the often discussed concept of “Ad-Hoc Process”.  Central to a case is the idea that there is more than one thing to do, usually requiring coordination of more than one person, so the term process fits.  The question is: what kind of process?

I believe the answer lies in the word “Routine“.   A routine process is one that is “well known”, “repeatable”, “normal”, and “common”.   The most important thing about routine tasks or routine jobs is that they are “predictable” – there are no surprises.  Case management is for dealing with processes which are not predictable in advance.

My proposed definition is then simply:

Case Management is the handling of non-routine work processes.

This will not replace the earlier lengthy precise definition for those who want to be technical or scholarly.  But for day-to-day use, this is a fairly accurate and useful definition of what Case Management is used for, without getting bogged down in specifics about how one might accomplish it.

One might argue that this is too broad and not precise enough, but it is hard to find examples of work that fit this definition and could not be considered case management.  Patient care, in general, is not routine.  Legal cases are in general not routine.  Law enforcement investigations are not routine.  The type of work that executives and knowledge workers perform are not routine either. It is a useful definition for that conversation in the elevator.

You might instead argue that the definition is too narrow, because there are “routine cases” which would be excluded, but I would point out that cases are routine only in hindsight.  Going into the case there is no way to know they are routine.  A routine task or a routine job would be one that would be predictable and well laid out what has to happen.

The concept of “non-routine” covers tasks that are not predictable.  It also covers tasks that might be predictable, but are very uncommon.  In business process management it is the case that there are some well known processes which are so rare, that it is simply not worth drawing up the process.  For example, filing a claim for a fire at the office might be a perfectly predictable process, but rarely needed, and so probably not automated.  It is likely that Case Management techniques will remain the best way of handling these kinds of work processes.

Comments?  It will be interesting to see if this parsimonious definition stands up to scrutiny.

So What is Dynamic BPM?

This is a related concept.  Fujitsu is using the term “Dynamic BPM” to cover the idea of a technology that can be used for predictable processes, as well as unpredictable processes.  It spans the range from System Centric BPM, to Human Centric BPM, to Case Management.

Find the complete talk, audio/video by Alistair Cockburn with slides on the InfoQ site. I am spending a lot of time on Agile Software Development techniques, and this talk presents a number of important concepts, so I am posting my notes here to help others looking for good materials on Agile software engineering.

Software Development Definition

Alistair defines software engineering as:

• People inventing, deciding, communicating
  • Solving a problem they don’t understand which keeps changing.
  • Creating a Solution they don’t understand which keeps changing.
  • Expressing Ideas in Languages they don’t understand which keeps changing.
• to an interpreter unforgiving of error
  • Making Decisions with limited resources, and every decision has economic consequences.

Cost of Distance

Distance is expensive.

• Say that two people sitting and working on the same screen, asking each other questions every 20 minutes is a cost of zero. Thus a 12 person team working this way would be the base “cost”.
• If people are in the next office with a wall in between, they have to stand up and walk next door, a 12 person team will incur an additional cost of $100K
• If people are located on separate floors of a building, a 12 person team will incur an additional cost of $300K, but more importantly, people will stop asking the questions which might be more damaging.

Learning in Three Phases

• Shu – learning a technique, starting by copying something.
• Ha – collecting techniques.
• Ri – invent, blend techniques.

Comparing Manufacturing and Software (Design)

• Software development can look like manufacturing if the unit of inventory is “Unvalidated Decisions”. Each line of code is a decision. All of the specifications are decisions. High level decisions are broken into lower level decisions, and programmers code the decisions ultimately into the product. Software development then is a factory to “process decisions”.
• Manufacturers hate feedback and rework. Doing something again is evil and wasteful, but this is not the case in collaborative design. Reworking is not a problem.

Lean Manufacturing

Want to aim for continuous flow. Avoid big waves of code. Small batch size. Nano-incremental development. If you are sending 1000 lines of code to test — too big! Instead, send daily integration to test.

Theory of constraints. Watch your bottlenecks.

Design is Knowledge Acquisition

Trying to learn about a domain, and trying to learn about a solution. When do you know most about the problem? The day after you deliver.

• Waterfall: there is a lot of typing, but the learning happens at the moment of integration. Learning everything at the end is bad. Waterfall is a late-learning strategy.
• Agile: growth of knowledge can be faster than cost because of early, rapid integration.
• Highest Business Value First – a sort of hybrid where you alternated between exploratory learning and then implementation.
• Trim the Tail – can decide toward end when to ship and whether to delay, because risk is low.

I will warn you, this is not a particularly interesting article — I just needed a place to record the results of some detective work.

I am a member of a number of “Groups” on “Linked-In”, one if particular:  Business Process Management Professionals Group.  I will be leaving that group unless something can be done about a particular type of spam.  There are these email notices that get sent about interesting articles that members link to.  But there is a problem.  Here is a screen shot:

What you are seeing above is four nearly identical links to a site called “Tickled by Life”, posted by four seemingly different people, but with the same unimaginative comment.  Who are these people?

shipra gupta – made comment on Aug 20 at 6:53am;  claims the title of “Independent Computer Software Professional”, has no history, has five connections, and is a member of 14 groups.

robin corbin – made comment on Aug 20 at 6:40am; claims the title “Independent Computer Software Professional”, has no job history, no connections, and is a member of 15 groups.

prashant gaur – made comment on Aug 20 at 6:18am; claims the title “Independent Computer Software Professional”, has no job history, no connections, and is a member of 17 groups.

Anthony Jude – made comment on Aug 20 at 6:06am; claims the title “Tickler at Tickled By Life”, has no job history, one connection, and is a member of 11 groups.

I expect this last title is the only correct one, only “Tickler” should be “Spammer”.

I don’t have any particular problem with the site — some of the articles are pretty interesting — but they have nothing to do with “BPM Professionals” which is why I joined that group.  I expect to get information relevant to BPM Professionals.

What we have instead is a flagrant attempt to increase the value of a site by making many many links to it, on many Linked in groups.  The above posts are made dishonestly by creating fake accounts, and making it look like the links to articles come from independent people, when in fact they are all from the same person/team.  The addition of these links causes me to get email  i.e. real spam.

What can be done? – Clearly a Linked-In group needs to be policed in order to be valuable.  There needs to be someone who actively kicks out people who join purposely for an agenda which has nothing to do with the goals of the group.  Whoever runs the group has to balance the desire for a large group, with the desire for a quality group.

A simple test for membership is to check the history on the person asking to join, and see if it matches up with real life.  This is work, to be sure, but like all things the value comes only from real work.

And where does this end up? Groups that are very hard to join, that screen carefullly the members to be sure that they are all who they say they are, and are most likely to not abuse the membership.  Groups will get more exclusive, and the best groups will be very hard to get into — and the ones that are easy to get into I don’t want to be a member of.

Paraphrasing the old adage from Groucho Marx: “I don’t care to join a Linked-In Group that’s prepared to have me as a member.”

I recently got an iPhone.  Cool piece of hardware and a lovely system design.  I was hoping that finally I could carry a single device for phone and podcasts, but what a disappointment!

There is a particular podcast that I like.  It is about 50 minutes long, and I have a 25 minute commute each direction.  The show appears on the web site about 2pm every day.  Sometime around 4pm I download the podcast, transfer it to the player.  I listen to the first half on the way home, and the second half on the way into work the next.  If I happen to be home at 4pm, I use my home computer, and transfer it to the player.  If I am traveling on business, I use my laptop, and transfer it to the player.

Can I do this with the iPhone?  Noooooooooo.  I get this:

You mean I have to delete all the FREE podcasts on my iPhone, so I can put one more FREE podcast on my phone?  What a pain!  Sometimes I retrieve podcasts at home.  If my iPhone is synced from home, then when I am at the office I am stuck — I have to drive home to get the podcast for the drive home.  Or, if I sync at the office, how then do I pick them up on the weekend?  Never mind picking up a podcast while on a business trip.  I could always sync to the laptop, but that is a company issued equipment and I can’t install iTunes on it.  This idea that you can sync at only one location is so broken.

So my friends point out: you can download directly to the iPhone!  So at 4pm I check the Apple Store and today’s show is not there.  It seems to take too long for the Apple Store to find out about the new episode; they only have yesterday’s show.

Research leads to conflicting reports.  WinAmp is supposed to transfer to an iPod, but other reports say the iPhone has tighter security that prevents this.  I have not tried it yet.  There are some sketchy sounding software brands offering solutions.  Other places offer solutions if I JailBreak the phone.  I thought the iPhone was supposed to make my life easier!   I am sure I will eventually find the right software. (suggestions anyone?)  All I want to do is listen to a podcast on my drive home. 

This feels like a brand new cool car, that you have to hotwire every morning.

I realize that synch-prevention is a feature of iTunes designed by the lawyers.  But why prevent synchronization of free content which it knows is free because you downloaded it from the Apple Store — for free?  And what about free content from Jamendo?

You can bet I won’t be giving up my MP3 player any time soon.  It is just sooooo much more convenient.