Monday, October 26, 2009

Comments on The IT Complexity Crisis: by Roger Sessions

Roger Sessions has written a paper on IT Complexity. When I met with Roger in London earlier this year (at the EAC) we had a short conversation, and it was clear we agreed on numerous things. This paper is useful because it provides a basis for more detailed discussion.

Roger’s paper:


  1. Cost of complexity is a little suspect to me. Primary issue is basing numbers on government data – government projects have high probability of failure, at least in the UK. More important I think complexity has more impact on full life cycle costs. Direct IT cost is one issue, but lost business opportunity is probably the major cost. Hard to estimate. However we can all agree, it’s a it’s a big number.

  2. I am always wary of “functional decomposition”. It’s notoriously imprecise. At CBDI we recommend a combination of capabilities and business types (data). The outcome of Roger’s example SIP analysis looks similar to what would emerge from structured business type analysis, but in my experience the latter method is more reliable. More importantly I am vitally interested in developing an architecture that is a) demonstrably stable where it needs to be (managing business types such as customer, product and so on) and agile by design where it needs to be (managing process behaviour, transient data and events). I accept this may appear to be a form of religious debate, and we may have to agree to disagree, but I am interested to have the discussion.

  3. But regardless of how you arrive there, Roger and I are completely in agreement on the need for components. The component must be completely encapsulated, own it’s own data store(s) and be the sole provider of owned services. Roger’s components appear to be capabilities. No problem there, because in the CBDI reference architecture there are options. But I would be looking to separate out layered behaviour so that business type components (stable) are separate from process and event components (unstable, subject to change).

  4. The computation of SCUs (standard complexity units) is interesting. Roger’s methodology will lead you towards larger components with more internal dependencies. What’s the internal architecture look like Roger? The dependencies are still there. I am not concerned if the number of dependencies is very high, PROVIDING the dependencies are all well formed, reusable services and operations. Then it’s simply a question of ensuring you have good management (life cycle and run time) systems, which we do know how to do. I am more concerned by poor reference architecture (bad patterns, convergence of behaviors that should really be separate, bad or non existent contracts . . . )

  5. Finally I put more faith in principles, patterns and reference architecture than in numbers. Numbers can lie, and they often do. Whereas patterns and reference architecture are the distillation of good (and bad) experience and provide intelligent people good guidance to become even more intelligent.

  6. At CBDI we have developed methodology and process for modernization. We see real demand for rationalization and modernization of existing systems (full life cycle costs). The essence of the approach is to a) create the SOA fa├žade and b) componentize. You might like to take a look at our method for CA Gen, it’s just one method for one environment, but it illustrates we [practice what we preach. There’s a slideshare there that describes.

Good stuff Roger, fully support the component approach. Happy to dialog in more detail.

SOA Manifesto Marketing Twaddle

Last week a small group of mostly vendor representatives plus a couple of analysts and authors came together last week in a seemingly self important manner to develop an SOA manifesto. Of course the primary reason for the event was marketing, and we shouldn’t be surprised that the result is superficial in the extreme. The primary worry I have is that they came up with a manifesto that actually confuses rather then assists in wider, deeper understanding.

CBDI members will know from our long standing research that the key to architecture, and SOA specifically, is to define principles that a) uniquely differentiate the architectural style and b) enable governance. I wonder how this group managed to omit crucial matters such as separation of provider and consumer, standards based and abstraction?

Meaningless marketing twaddle, but because the vendors sponsored this, political correctness will prevail. Yuck.

The Death of Enterprise Architecture?

- Introducing Smart Ecosystem Architecture (SEA)

In recent months there has been considerable debate about Enterprise Architecture or EA. Practitioners who have embraced EA are now hotly discussing whether EA is a business or IT architecture and the balance between the two domains. Without any doubt most EA today is IT focused even though practitioners know it shouldn't be. And the new EA standard TOGAF 9 merely reinforces this. Although there is a vocal lobby who would like to gain more power and influence in the enterprise by taking the lead in business architecture and design, typically they are unable to achieve this because they can't articulate or demonstrate why the business should let them.

Yet my own observations are that EA has not been an overwhelming success. The tensions between EA and delivery teams remain undiminished. EAs are frequently out of touch with delivery issues and do not command the respect needed to exert strong governance over the delivery work streams. From an SOA perspective, there are few enterprises that have delivered a comprehensive Service Portfolio Plan, let alone the implemented enterprise service portfolio. You may wish to ponder on whether this is a failure of EA or SOA. I couldn’t possibly comment!

Many years ago when we published the first CBDI Maturity Model we recognized that SOA maturity would be defined by the ecosystem. This maturity stage followed the enterprise stage. I will readily admit that in the early days the ecosystem stage was ill defined; in fact no one was interested. Apart from a small minority of enterprises that have always operated an ecosystem business model, the focus of attention was always heavily on the enterprise. Today we can see things have moved on apace. Various influences particularly Complex Event Driven Architecture and Smart Business and IT are strongly predicated on optimizing business design and processes involving all the ecosystem stakeholders. Examples:

- Airlines, airports, airport concessions, airport services, hotels and rental car companies form an ecosystem that optimize their resources and prices on a dynamic basis dependent upon external events, raw demand and actual traffic. Actually we wrote a research report around this model years ago – see reference.

- Smart power grids with tighter linkage between customers, suppliers, generators, assets and operations manage power supply to optimize use of green energy, price to consumers and profitability.

- Also see IBM's Smart Planet site. Reference below.

The emphasis on business optimization is very interesting. This means that conventional architecture domains of operational systems, business intelligence (BI) and management information (MI) become much more inter dependent and dynamic. New types of information, perhaps conventionally classed as operational, now become critical MI. Real time feedback loops require derived information to be available in the operational timeframe. Conventional (sic) SOA layered architecture policy requires extension to manage new patterns.

In these examples of the new enterprise each participant collaborates to optimize the overall ecosystem and in the process optimize their own position vs their collaborators. In this world the architecture is absolutely not enterprise wide. It is goal driven, focusing on a scope that is business results driven.

Does this mean that EA is dead? As discussed, in its current form EA has actually been less than successful. The current level of debate is merely confirmation of the patently obvious. Personally I recommend a more practical approach to architecture that is more grounded with stronger relationships and shorter feedback loops to business value and delivery projects. Given the current issues with EA it would be expedient to have unambiguous naming without the baggage of EA. I am temped to say that Business Architecture works because most grownups understand that Business and IT are indistinguishable at this stage of the game. But I see a lot of immature debate from EAs that want to elevate EA away from IT, which is plainly wrong. Also I don’t have a strong opinion on the "role" of EA. Rather I worry about the nature of the architecture. I want to get everyone thinking about a new perspective and scope.

On balance therefore I prefer a naming that emphasizes the architecture scope and deliverables, the differences from what we have been doing and the need for change. I suggest Smart Ecosystem Architecture (SEA).


IBM Smart Planet

CBDI Report: Modeling for SOA - Worked Example - The passenger departure process - from arrival at the airport to boarding the plane.

CBDI Report: A Web Services Maturity Model

Thursday, October 1, 2009

The Failure of SOA Standards!

You will have heard the joke before: “The nice thing about standards is that there are so many to choose from.” Sadly in SOA land this is reality.

The need for standards in SOA are multivarious including standard concepts so we all speak a common language, standard meta models that underpin profiles, repositories, deliverables, specifications etc. standard protocols that allow implementation and platform neutrality for service publication and consumption. In this note I am interested in concept standards that support architecture and the full life cycle.

In this area we all know we have three “horizontal standards” organizations (OASIS, The Open Group and OMG) that have each followed their own path, ignoring the blindingly obvious need to have consistency across their work. The situation is so ridiculous that the three organizations resorted recently to publishing a document to explain how to navigate around the SOA architecture standards. Curiously although the document is branded by the three organizations and acknowledges participation and contribution from all three organizations, the document is formally a joint OASIS and Open Group document. One might be forgiven for assuming they couldn’t agree the legalities of the collaboration. Also the document doesn’t address inconsistencies between standards and guidance produced by the same organization.

The document declares two goals – 1) to help users to understand the strengths of each body of work and select products appropriate for their needs. 2) to encourage consistency. But the latter is declared only as a secondary goal!

But inconsistency is only part of the problem. We must judge any standard in its support for the user community and crucially user adoption. The primary purpose of these standards under discussion is achieving common definitions of important concepts and simply put they have failed to achieve widespread adoption because of the competing efforts have confused the user community. Further individual efforts have in my opinion failed to meet user needs.

  • While TOGAF is getting considerable traction our experience is that the parts of TOGAF that get used are the ADM and the contracts. Not the concepts and meta model, because they are patently not based on real world experience. They are both too verbose for some purposes and insufficiently detailed for others. And the guidance for use is non existent.
  • Similarly the OASIS model is widely regarded as academic – not useful in practical application.
  • I have more respect for the OMG’s work in SoaML, because it is usable in support of code generation, but it suffers from over generalization, which makes it very hard to use by mere mortals.

I observe that in the referenced “navigating” document the nearest the authors come to agreements on core concepts is at the most superficial level. I see no evidence that there is any real intent to deliver real convergence at a meaningful (i.e detailed, defined) level of abstraction.

Over the years CBDI has encouraged standards efforts; we have consistently made the case that common concept standards are essential at all levels of abstraction. We didn’t just sit by; we developed our own meta model which actually predates most of the work referred to above, and made it available to standards bodies. Frankly we fully expected that our work would be superseded by the standards work and that we would progressively align our SAE models to a converged standard.

Well you won’t be surprised to hear that the complete opposite has happened. This month we are publishing V3 of the CBDI-SAE Meta Model for SOA. We have actively participated and contributed to the OMG SoaML work and we are pleased to announce a high level of alignment. This is possible because of the highly generic nature of the SoaML work.

The CBDI-SAE meta model has always been very different to the various standards bodies work. We have focused on what may be termed a detailed Business Type Model (DBTM). The goals of the DBTM are to provide consistency of concept for use across the life cycle at a level that can be implemented in tools and inform practical architecture and full life cycle deliverables. And of course users of the CBDI-SAE meta model tell us that is exactly what they use it for – populating architecture and service repository schemas, underpinning UML profiles and defining detailed of deliverables that can provide full life cycle traceability and governance.

What we didn’t anticipate at the outset was that our meta model would actually form the basis for a further purpose – to provide a mapping between the other concept standards such that users forced to work with multiple conceptual schemas (few of us live in a homogeneous world) would have a basis for coordination and possibly transformation.

So as a user what do you do? Currently we see minimal use of these concept standards because they are inadequate. We recommend that you continue (or commence if you are a late adopter) using the CBDI-SAE meta model. We will commit to map to the other standards and, if and when they ever get to a credible position of convergence, we will certainly plan model convergence also. But we will need more than alignment of the concepts, we will need to see practical abstraction and detail required for real work.

We will be announcing a user review of the V3 model very shortly and welcome all input.

Navigating the SOA Standards Landscape Around Architecture, Copyright The Open Group 2009, OASIS 2009,

V3 SAE Meta Model note this is currently for CBDI subscribers only. Public domain version follows shortly for the review.