When Mandelbrot met Lynch (The Recursive Enterprise, Part I)
In most of my previous posts I have described nowadays Enterprise IT as a holistic and organic system. From an IT macro-perspective, all business processes, business applications and the hardware that supports them are hubs and nodes in a multi-dimensional, highly complex and integrated graph, serving the business needs for information storage and retrieval.
What's amazing in this story is that IT macro-perspective is identical to an IT micro-perspective. If you take, for instance, a single business process, you'd discover that it is a mini-IT, spanning across multiple applications, hardware and (even) organizations, very similar to the multi-dimensional and complex graph of the entire IT. Differently put, "zooming in" from the Enterprise level to more internal levels simply shows similar pictures.
When macro perspective matches micro perspective we think of fractals. "Fractals graphically portray the notion of 'worlds within worlds' which has obsessed Western culture from its tenth-century beginnings."
Fractals are found in abundance in organic natural objects like clouds, mountains, coasts lines, river networks, leaves, dust, systems of blood vessels and so forth. Organic systems are mostly nonlinear and chaotic.
So Enterprise IT is a chaotic system and long-term prediction about the data center service level is irrealistic (IT is un-QoS-able), just like any attempt to have a long-term prediction of the weather is futile.
Enterprise SOA is aggravating the situation. By facilitating an easy assembly of Services into new virtual applications, Enterprise IT is becoming fractalized and Recursive more than ever before. And pinpointing problems in normal daily operations is… simply different.
Take, for example, a Business Process encapsulated in a Service. Customer Service Support complains that they get too many calls regarding the performance of this Service (that's the typical SOA "Systems are slow" fault, which, as you'll soon see, can drive you crazy). There's nothing suspicious in the IT ops monitoring console, so some brave IT people decide to have a look inside the problematic Service. And what they discover inside are... more Services; they peer into each of these – and the picture's the same: more Services! In three years or so, they'll get to the last functional Service just to reveal some more service calls, this time to the GRID management layer, asking for Computing Resources. The GRID management layer will invoke the Services of the virtual shared-distributed memory, of the virtual SAN and of the virtual machines. The VM layer will call the remote GRID service made available by IBM, to get drops of CPU leftovers from their facilities located on the dark side of the moon. And so forth.
Trying to pinpoint the exact place of a problem is like finding a needle in a hay stack. I can envision some high ranking IT Officer shouting, enraged: "Stop with these Services! Bring me R-E-A-L ___ing Objects!!!" and some will vehemently swear back: "We'll not rest till we find it, Sir!". But, just like the Roman soldiers who flooded the small hiding room of the Judian People Front (or was it the People's Front Of Judia), these IT guys will eventually come back, humiliated, whispering "We found this", raising a spoon up high. Indeed, a "Bigus Dickus" situation.
This situation can be avoided, with the help of the Log Lady and some of her friends (But I'll stick to logging in this post).
Following our fractal thinking, we would expect to find similar patterns in both the micro and the macro levels. So if a micro-object, like an Apache Web Server, is logging the well-being of its components, we'd expect to see similar logging patterns on the Enterprise level as well. Enterprise components are business processes, products and services. We'd like to see than, for a start, a detailed log of a business process.
But think about it for a second: what is the meaning of logging an SOA, distributed, asynchronous, Mini-IT, business process? Is it the collection of all logs produced by each of its sub-components? But these n sub-components are serving, not just one business process, but rather hundreds or thousands of them. So how would we know which log entry in each of the sub-components is related to our business process in question? That's a colossal correlation challenge.
Or should we conceive a new logging technique for the purpose of our new fractal, recursive, Enterprise?
In my next post, I will discuss the current state of Enterprise Logging and we'll see where it will get us.
"Stars, moons, and planets remind us of protons, neutrons, and electrons". the Log Lady.