On Metadata-Driven Analysis Management Objects (New Link)

All code listings described in the following posts on Metadata-Driven AMO (Parts 1, 2 and 3) may now be accessed at the link below...



Cheers to Windows Live SkyDrive. Jeers to Ripway.

- Adrian Downes

On the Gathering Clouds of Software as a Service (SaaS)

Recently, Becky Nagel at RedmondMag.com broke a story about the next step in Microsoft's Software as a Service (SaaS) strategy: Microsoft Online Services.

For those of us building solutions in the Business Intelligence space, and haven't already spared a thought about SaaS, it is important to be consider of the mid-to-long term implications of a hosted system-of-record (S-OR) application.

Think about the major ERP vendors moving their product to a hosted, SaaS offering (like the SAP BusinessOne trials in India in 2007). In this business model, one customer may leverage a hosted ERP, for example, paying a monthly or annual fee to use only CRM and Financials services, whilst another may prefer to use additional services for managing Suppliers, Inventory, Employees (etc.). Quickly, the third-party hosted provider is able to leverage (potentially) incredible economies of scale.

Basically we are talking about a web-based “data-in” proposition, where a business collectively enters transactional data securely via the browser for one or more key business functions. On the opposite side of the fence, BI fundamentally is a “data-out” paradigm where business data is refined and exposed as information for decision making (either strategic, operational or even individual).

Now, some of you may recall the famous book and articles by Nicholas Garr (Does IT Matter, Harvard Business School Press, 2004), who discusses the commoditisation of IT into a utility of services (aka. utility computing). Naturally there was something of a backlash from IT Managers/Directors, CTOs, and CIOs (three guesses as to why). This is what SaaS is all about and where many in the IT industry (Gartner, Forrester, IDC, and obviously the major vendors) believe the trend is heading.

From a BI Practitioner perspective, there is certainly food for thought:

First, consider the S-OR application to be, in general, the ideal data source for our BI solutions since such systems typically encompass multiple business functions or domains (again, Customers, Financials, Suppliers, Employees, Inventory, etc.). The S-OR is the end goal of a Master Data Management effort, and (as I'm sure must may currently/historically agree) often the oasis of many data integration (ETL) efforts.

Next, consider our accessibility to S-OR applications should they become hosted (or “in the cloud”). Suddenly, it seems building a BI infrastructure on-site for a customer becomes somewhat less of an issue. Right? "Cool", you might say, "web-based BI" or "hosted BI".

Now consider the shifting burden of information from a customer environment to a third-party hosting provider. Do you think it will be easier or more difficult to access data and build those lovely BI solutions for customers?

Finally, consider this: if utility computing takes root, and the hosted system of record becomes a feasible and sustainable model for business, guess where BI is going next? More profoundly, who do you suppose will be in the most likely position to deliver this likely "new form" of BI?

What are your thoughts?

- Adrian Downes

On Preparing for the MCITP: BI Developer Certification

Kudos to world-reknowned consultant, trainer and blogger Teo Lachev, founder of Prologika and author of Applied Microsoft Analysis Services 2005, on his latest efforts. A new training guide, MCTS Self-Paced Training Kit (Exam 70-445): Microsoft SQL Server 2005 Business Intelligence—Implementation and Maintenance is designed to help you prepare for the first of two Microsoft certification exams towards the MCITP: Business Intelligence Developer designation. The guide is now available from Microsoft Press and can be ordered on Amazon.

Although a book like this is certainly long overdue, I expect it to be a huge success-- I happen to know more than a few BI developers and consultants out there eager to pick up this new title. Congratulations to Teo (who provided the SSAS 2005 content) and his co-authors Erik Veerman, Dejan Sarka, and Javier Loria.

- Adrian Downes

On "Mythbusting" Business Intelligence

It is always refreshing when someone steps up and challenges the status quo when it comes to identifying just what BI really is, and what it means to organizations of all kinds. Jamie Thompson's excellent blog (and a recommended read if you aren't subscribing to it already) sets out to bust up a few myths in BI, in a recent July 18th post.

Cheers, Jamie.... thanks for the healthy wakeup call on current BI (mis)perceptions!

- Adrian Downes

On Metadata-Driven Analysis Management Objects (Part 3)

Enhancements to the CreateNewAggregationDesign Function

Most BI practitioners will be taking full advantage of multiple-measure group support in SSAS 2005 cubes (UDMs) for a variety of reasons including support for one of the many-to-many dimension design patterns eloquently positioned by Marco Russo in his work "The Many to Many Revolution". I have two successful implementations which build on his great ideas, and, it was during this phase of development that I came across an opportunity to enhance our code base for our modest Metadata-Driven-AMO “quiver of arrows” (to quote a good friend of mine).

In order to take better advantage of multiple measure-group support in SSAS 2005 cubes, using the code listings from Part 2 of the Metadata-Driven AMO posts, you will find that it is far more appropriate to leverage the MeasureGroupDimension class instead of the CubeDimension class when adding AggregationDesignDimension objects to the .Dimensions collection of the AggregationDesign object. As you may recall, we use the AggregationDesign object to define which aggregations we will store in our cube, based on counts of dimension key attributes found in the incoming data for our new partition. The reason for this change is simple: as you iterate through a collection of CubeDimension objects, you may come across a perfectly valid CubeDimension which is not associated to the measure group partition we are generating. By using the MeasureGroupDimension class, you ensure that you are only creating aggregations relevant to your measure group partition. Moreover, both the MeasureGroupDimension class and AggregationDesignDimension class share the same inheritance lineage, so we are able to take advantage of the .CubeDimensionID property without any fuss. In Listing 10, the minor changes can be found in Section 4.

Update (14-MAY-2008):
All Listings described above may now be accessed at the link below...



- Adrian Downes

On New Thinking in Service-Oriented Business Intelligence

Today one of my clients was kind enough to call my attention to an excellent article published last month by Arnon Rotem-Gal-Oz, which briefly reviews SOA and BI (ETL, specifically), considers the challenges in trying to resolve a service-oriented business intelligence (SoBI) architecture (network bandwidth costs from polling a service), and, positions the push model of event-driven architecture (EDA) as a way to overcome said challenges. The article is definitely worth the read, particularly as challenges (there's that word again) to conventional thinking on ETL emerge in increasingly service-oriented environments.

- Adrian Downes

On Three Perspectives of Performance Management

Not too long ago, I was challenged by someone who was adamant that Microsoft Office SharePoint Server (MOSS) 2007 was all that organisations needed for decision making, particularly in light of its Business Intelligence feature set. The main questions I receive from customers attempt to reveal how adding PerformancePoint Server 2007 to their existing operational environment (ERP, CRM, POS) can enable performance management for their businesses. Being a Microsoft BI advocate I normally point out, in both cases, the importance of a consolidated data tier, one which centralises operational data from a number of sources into a company-wide "single version of the truth". Unsurprisingly, I discuss SQL Server 2005's "BI Services" as the means to refine raw line-of-business data into useful information.

Lately, when I reflect on performance management as a business discipline, I find that my thinking has changed somewhat.

Consider the following value propositions:

The Data Management value proposition
A known quantity in of SQL Server 2005 "BI Services" in terms of centralising and surfacing structured data as KPI values and other measures to BI applications

The Information Management value proposition
Involving, through MOSS 2007, the optimisation of document and content control (un/semi-structured information) with human access and workflow/business process interactions.

As always I tend to relate in terms of Microsoft technology, but feel free to substitute specific technologies mentioned with those you work with...

If performance management really intends to align corporate strategy with information, people and processes, then both "data management" and "information management" capabilities need to be considered-- simply proposing a PerformancePoint & SQL Server solution in support of Strategy Map and KPI shape business requirements is not enough for a performance management initiative.

The art and science of performance management has many incarnations spanning well over 100 years of industrialisation and automation, arguably starting with the formal positioning of Scientific Management (1911), which reflects the ideas and concepts put forth by Frederick Taylor (1856 - 1915). Taylor built on earlier companies-as-machines metaphors, and introduced time-and-motion studies which attempt to uncover optimal performance in work processes (both human and early machines). In the Information Age of the 21st century, the theme of aligning technical and human domains with strategic direction can be found in a number of performance management methodologies:

Performance Pyramid (McNair, Lynch & Cross, 1990)
The Performance Pyramid approach views organisation as having four interdependent levels, specifically: corporate management, business unit, processes germane a given parent business unit (such as those geared toward customer satisfaction or market share) and operational goals which support a given process. At the operational goal-level, values such as time and quality are determined at different frequencies and used to meet management requirements at higher levels.

Note: some publicly-accessible references to the Performance Pyramid only cite Lynch & Cross

Effective Process/Performance Measurement (EP2M) Model (Adams & Roberts, 1993)
EP2M also views an organisation as having four taxonomies of measurements: top-down measures which are used to manage strategy and change, bottom-up measures which consider human action and the outcomes of ownership and accountability, internal measures which are used to improve and sustain both process efficiency and effectiveness, as well as external facing measures geared towards markets, customers and suppliers.

Balanced Scorecard (Kaplan & Norton, 1992)
Probably most familiar in business circles these days, the Balanced Scorecard is a framework that can be applied (and modified if necessary) to suit critical perspectives of a given organisation. Typically four perspectives are employed: financial, customer, internal process and learning and growth. Each perspective represents a high-level collection of relevant key strategies, objectives and KPIs, and, is often represented visually with tools like the Strategy Map.

In each the methodologies listed, process and human factors (and, strangely, the number four) play important roles in organisational performance. It is becoming an increasingly accepted practice in business to consider such factors, alongside traditional financial measurements, in order to arrive at a more complete organisational picture. It follows that both factors become passive organisational elements subject to measurement, and, active elements for organisational communication and collaboration across performance management cycles.

Indeed, it would seem that effective performance management comes about at the intersection of corporate strategy (fleshed out by a suitable performance management methodology), data management (providing structured data for decision-making) and information management (representing unstructured content, semi-structured processes and sometimes pretty complex people). Moreover, encouraging the use of MOSS 2007 alongside SQL Server 2005 and PerformancePoint would probably prove more valuable in the long-run (albeit more expensive, unless Windows SharePoint Services 3.0 is considered) to a performance management initiative.

Thanks for reading! How would you rate the value of information management in performance management?

- Adrian Downes

On Metadata-Driven Analysis Management Objects (Part 2)

Part 2: Code to Create and Process the Partition

In order to take advantage of AMO we naturally need to make use of the classes available in Microsoft.AnalysisServices.dll. Once we have defined the variables (discussed in Part 1) within the SSIS package, and wired them up to the single-row output from our metadata store using an Execute SQL Task, we can reliably retrieve and use them within the package Script Task. Our script task uses three files:

• ScriptMain: serves as an entry point, connecting variable values with PartitionGeneratorController

• PartitionGeneratorContorller: class file which governs execution of a PartitionGenerator object

• PartitionGenerator: class containing all the properties and methods for creating and processing partitions and aggregations

A reference to Microsoft.AnalysisServices is made, ensuring that the latter two class files can use AMO appropriately.

Listing 1 is a snippet within ScriptMain, showing how the variables values are captured from the incoming DTS.Variables collection.

Next, we use the controller class to field the incoming arguments, and supply them to our PartitionGenerator object, which then creates a connection to the server (supplied by OLAPServerName and OLAPConnectionString) and iterates through collections of databases, cubes, measure groups and finally partitions. We use the metadata supplied to help us mine our way from collection to collection. Essentially, at the leaf-partition-level, the incoming partition name will always be new, since we concatentate a new DateKey to the PartitionName prefix. As mentioned in Part 1, this approach follows a "new partition each day" approach for both loading new data into the relational fact table as well as the target cube measure group. Listing 2 shows part of the nested-loop logic.

Once we have determined that the incoming partiton does not exist, we create a new partition with PartitionGenerator's .CreateNewPartition function. In Listing 3, arguments for the function are supplied as current SSAS 2005 objects we identified along the way through the nested For Each loops.

PartitionGenerator uses the QueryBindingString metadata value as an argument to define the QueryBinding property for the new partition. An object argument for the measure group is applied since it contains a reliable .ID property value that is used to define a new partition within an existing collection, within an existing measure group. We are also sure to use the Cube.Update method to ensure that the new partition is saved to the target cube measure group. Listing 4 provides the complete code for the function .CreateNewPartition. Back in the controller, provided .CreateNewPartition returns a new partition object safe and sound, we can go ahead and process it (shown in Listing 5, this follows on from the controller code shown in Listing 3).

Using the method .ProcessNewPartition, the PartitionGenerator object accepts the newly created partition, along with the measure group and cube objects This method uses the ProcessType.ProcessData enumeration option, to optimise "fast-loading" of the new partition. Using ProcessData means that we only process the fact data for the partition, and does not handle any dependent objects. A separate step (not discussed in this series of posts) handles dimensions separately. Listing 6 provides the complete code for .ProcessNewPartition. In our testing, we found that we needed to separate partition and aggregation processing logic into discrete operations, delineated by Cube.Update operations, in order for the SSIS Script Task to work effectively. The beneficial by-product of using ProcessData to load our partition is that we can quickly process aggregations with the ProcessType.ProcessIndex option.

In order to create and process aggregations, PartitionGenerator exposes another function .CreateNewAggregationDesign, which uses a dynamically-generated partition count document stored in the variable OLAPPartitionKeyCounts. This document describes the alignment between a fact table key column and its corresponding cube dimension attribute found in the partition. The PartitionCount attribute defines the distinct counts of dimension key members from the newly-generated relational fact table partition (this is also handled by a separate process, and is not covered here). Listing 7 shows what an example PartitionCount document looks like.

Further in the controller, provided the new partition is created and "fast-loaded" successfully, .CreateNewAggregationDesign is called (see Listing 8), returning a new AggregationDesign object driven by PartitionCount. The AggregationDesign is resident within a measure group, and is applied to the new partition; the ProcessType.ProcessIndexes enumeration is used to build the aggregation data and bitmap indexes for the new partition.

Inside .CreateNewAggregationDesign, the following steps are taken:

1. Determine whether an AggregationDesign already exists within the measure group (if it does, this represents a condition where the process itself is being re-run, thus the existing AggregationDesign object is overwritten by a new one with the same name)

2. Specify the EstimatedPerformanceGain property value for the AggregationDesign object (used for annotation purposes, this accomplished via the OptimisationLimit metadata value)

3. Specify the EstimatedRowCount property value from the PartitionCount document, in order to the specify the total number of rows for the partition

4. Add AggregationDesignDimension objects to the .Dimensions collection of the AggregationDesign object

5. From the PartitionCount XML document, set the EstimatedCount property value for each specified AggregationDesignDimension attribute

6. Design Aggregations for the AggregationDesign object, using the OptimisationLimit value, and return the object.

The full listing for .CreateNewAggregationDesign is shown in Listing 9. Note that we only use the referenced (ByRef) variables isFinished and optimisation in the .DesignAggregations method (of the AggregationDesign object) to indicate the limits on designing aggregations. The final step is analagous to working with the Set Aggregations Options dialog in the Aggregation Design Wizard, prior to clicking the Start button.

That's pretty much all there is to it. We now have a package in place which, with the right metadata, can process any partition on any cube measure group. On examining the completed partition (either via SSMS, or directly in XML), all the annotations, aggregation designs (visible at the measure group level), and aggregations are present. It should come as no suprise that our client queries tested significantly faster with the aggregations over no aggregations at all.

For those of you eager to rip the Listings to bits, bear in mind that a few alternates were considered before this final approach. One of the alternatives to applying the .NET code in an SSIS Script Task would be to create a stored procedure that calls a PartitionGenerator assembly (wrapping both the core class and its controller). Both Microsoft.AnalysisServices.dll and PartitionGenerator.dll would need to be registered, and, you would need to assign the permission set UNSAFE. On paper this is fine, but when you try to register Microsoft.AnalysisServices in the Database Engine, you may receive the following message in Management Studio.

Warning: The SQL Server client assembly 'microsoft.analysisservices, version=9.0.242.0, culture=neutral, publickeytoken=89845dcd8080cc91, processorarchitecture=msil.' you are registering is not fully tested in SQL Server hosted environment.

The risk conveyed by the warning was sufficient for us to explore alternatives. By the way, trying to register version 9.0.3042.0 of the .dll from SQL Server 2005 Service Pack 2 returned the same message in our environment.

Other serialisation issues also surfaced as we tried to iterate through collections of AggregationDesign objects. Such issues may be addressed by decorating one or more of the classes and methods with the Serializable() attribute in PartitionGenerator, although, since AMO is leveraged behind the scenes, it follows that they would likely need to expose classes that are serializable as well. Unfortunately, we did not have the time to research this any deeper, but I would like to hear if anyone else has found success taking this approach.

Other solutions, such as creating an SSAS assembly, or passing ASSL (XML/A) statements (either via an assembly or through a script task), did not satisfy our 'ease of support' requirements as mentioned in Part 1. I'm quite certain, however, that dynamic creation and execution of ASSL would work nicely.

While it does not profess to be the magic bullet solution, I hope it proves helpful to anyone coming to grips with AMO. As always, I welcome your feedback; if you have found a better or more efficient way around some of the issues raised, then please feel free to share!

Update (14-MAY-2008):
All Listings described above may now be accessed at the link below...



- Adrian Downes

On Metadata-Driven Analysis Management Objects (Part 1)

Part 1: Background

Over the last few months we have been hard at work on a metadata-driven ETL pipeline for an 880Gb SQL Server 2005 BI solution. Powered by SSIS, metadata has allowed us to create a single SSIS package for a generic task, re-using it many times based entirely on the information we supply to it.

For instance, our solution has a single "Import Flat File" package which uses metadata describing the source file, the system from which it originated (for traceability, among other reasons), the target "Import" table as well as a destination "Archive" folder for the file itself, once the package completes.

We take the same approach for performing surrogate-key generation, referential-integrity checks, data transformations, loads, archives and some maintennance operations in the solution.

At a higher level of abstraction, we use process codes as both an entry-point to describe all metadata associated to a given process as well as the foundation for a point-in-time instance of execution for said process (the latter is important for logging process events and errors). Metadata allows us to semantically "chain" together streams of Extract (the "Import" package described above), Transform and Load processes (as well as other supporting processes). We are able to govern the execution of process schedules through an external application, which only needs to execute dtexec (a command-line utility for executing SSIS packages) with the appropriate SSIS package and process code argument. The external process governor allows us to execute discrete processes either sequentially or in parallel, in-line with precedence-constraints for loading target tables in an Inmon-style relational data warehouse. Our relational fact tables are all partitioned, supporting high-volume daily loads, and, the generation of partitions is also metadata-driven.

Further along the value-chain, we have an SSAS 2005 database which houses a single cube (UDM) supporting a number of measure groups. Each of the measure groups is front-loaded by day-level partitions, allowing our users to continue working with their desired perspective while the new data is being processed in. The measure groups are aligned to corresponding relational fact table partitions by (you guessed it) metadata.

Altogether, the SSIS facet of the solution operates on just 8 small packages. Essentially, if there are any errors raised by the process- governing schedule by a problematic process, the problem is 99% of the time metadata-related. More to the point, we continue to add value to the system, by only implementing new structures (either in the relational or multi-dimensional areas of the solution) and metadata to describe it along established standards.

In keeping with our approach of metadata-driven reusability, we created a single package to process a new partition for any measure group, driven by a process code, and group of related metadata describing:

• ServerName (the target SSAS 2005 server instance)
• DatabaseName (the target SSAS 2005 database within the server instance)
• CubeName (the target UDM within the SSAS 2005 database)
• MeasureGroupName (the name of the measure group, which has metadata mappings to a corresponding relational fact table)
• PartitionName (stored as a partition prefix in the format "MeasureGroupName_" which is concatenated to a DateKey to describe the "slice" of data within a given partition)
• DateKey (an 8 digit integer, also known as a "smart key" in the format YYYYMMDD, and used as the basis for relational fact table partitioning)
• QueryBinding (the query string used to define the dimension attribute keys and measure members for a measuregroup, sliced by the DateKey)
• ConnectionString (relevant for dynamically connecting to a given SSAS 2005 database at run-time, providing future-proofed flexibility if another SSAS 2005 database needs to be implemented)
• PartitionKeyCounts (an XML document which describes the association between a dimension key column in the relational fact table and the corresponding cube dimension attribute in the UDM.)
• OptmisationLimit (reflects the "Performance Gain Reaches" property in the Aggregation Design Wizard)

This collection of metadata allows us precise control over which measure group is to be front-loaded with a new partition, and provides a high degree of flexibility for multi-server, multi-database scenarios as they may emerge.

The high volume of data the system is subject to, and, the demand for high query performance warrants precise counts of partition fact keys. Thus, in order to dynamically build and process our partitions, in addition to building and processing our aggregations at run time, we looked to AMO for simplicity in handling metadata. A generic SPROC, called by the SSIS package in an Execute SQL Task, with the appropriate process code would retrieve the "OLAP Partition Processing" metadata described above as a single row result-set, mapped to package variables, and supplied to an SSIS Script Task. We also chose to look beyond the stock control-flow tasks (Analysis Services Execute DDL Task, Analysis Services Processing Task) and data-flow tasks (Partition Processing Task) in order to ease supportability for administrators since they already understood VB.NET over ASSL (XML based Analysis Services Scripting Language).

In the second installment of this post I will show some of the code samples used to leverage AMO, and the metadata supplied to it, in an SSIS Script task.

- Adrian Downes

On Market and Product Consolidation

It should be pretty old news by now concerning the acquisition of Hyperion by Oracle and Hyperion's acquisition of Brio a few years earlier. While the cited article reports Business Objects position on the recent deal as "(creating) a bit more confusion in the marketplace", it is such market and product consolidation which works best for consumers looking for a complete business intelligence product, designed to support a performance management initiative.

Consider Microsoft Office Business Scorecard Manager 2005, for example. BSM 2005 currently supports monitoring, and analysis tasks through scorecard and report views displayed to users through SharePoint. However, to complete the remaining tasks in a typical performance management cycle, further off-line work would be required with additional tools like Microsoft Office Excel 2003 (or later) to assist with planning / budgeting tasks. Subsequent forecasting could be supported, with more work using SQL Server 2005 Analysis Services data mining models. Some have actually extended BSM 2005's data visualization capability by integrating with ProClariy Analytics helping users to better understand what's happening behind a given performance measurement or KPI.

The Microsoft example notwithstanding, the net effect of having multiple BI applications involved for a specific need becomes risky and problematic due to issues such as:

• potentially incompatibility among certain BI applications, from different vendors
• duplication of effort among BI applications
• increased licensing costs for separate BI applications

In SQL Server 2005 such problems don't exist, since integration, analysis, and reporting functions are consolidated into the single product. Consolidation is a great thing from a consumer perspective, since it solves the BI application issues (above) and provides increased value-for-dollar for the product itself. Moreover, the growing movement towards performance management demands an application that addresses each of the discipline’s tasks or phases.

The very idea that such aggressive consolidation in the increasingly competitive BI software marketplace is occurring simply means that we as consumers will benefit from a push for better features, bundled into a single offer (regardless of the vendor), and ultimately lowering the total cost of ownership on our part. Most vendors now realize that the way to deliver performance management effectively is through an investment in a BI system. Microsoft reported in their announcement of PerformancePoint Server last June that they have been listening to customer issues and responded with a bundled approach of their own, rolling together monitoring and analysis with planning, budgeting and forecasting functions in a single product.

While some may speculate whether the Oracle play is a "me-too" tactic or a defensive posture to retain existing Oracle customers, the bottom line that change is happening-- and change is definitely good for us with performance management needs.

What do you think? Is the current state of the BI software market a good thing for consumers? Or, is it all becoming too confusing?

- Adrian Downes

On Versatility and Being "Business Intelligent"

For some time I worked with a small BI consultancy (back when it was called a Data Warehousing consultancy). I had recently transitioned to this company from a prior role where data warehousing was more of a pastime than anything else. Thus, I was excited that BI would become a full-time pursuit. What I would come to learn from this relatively small consultancy was that the percentage technical work was itself less in proportion to the business case assessments and project management tasks involved in a given BI project. I was a young hot-head at the time, and found myself often on the defensive in light of ambiguity and the seemingly changing requirements and priorities mid-stream from project to project. I found difficulty in understanding the financial and operational terms used by our customers, and, also found myself at odds with a certain project manager who seemed to be more interested in saying "yes" to each and every change request without consulting the technical consultants (myself included). By the time I was ready to move on, I had a gutful of what I believed to be "corporate B.S.", and felt that I was best served to "stay technical". On my last day, I ventured over to head-office to hand in my laptop. The managing partner asked me aside, and mentioned that he was aware of my displeasure with how certain projects were (mis)handled. Instead of asking further questions, he simply said:

"...to be in business intelligence, one must be 'business-intelligent'... even though things didn't work out for you here, it's important for you to realize the value of understanding the customer's business domain, as well as more about project management in order to understand how to effectively argue for or against change in cost-related terms. You might want to consider taking a course in project management or learning more about business in general"

Ah, what did he know? I thought defiantly.... surely he wasn't out on the front lines and didn't endure what I had been through. At the time, I concluded that staying technical was the way to go.

Years later, I worked with a larger IT consultancy that prided itself in employing depth specialists in certain areas of technology. The spoils of war for the company's directors were terrific for a time, but as the company grew they realized the benefit of having more people on staff with project management and business analysis skills. This seemed like a good idea, but the bottom line of the direct costs of labour forcibly reduced the profit margin for the directors, since the non-technical specialists were typically as expensive as the technical resources, but, were not billing proportionately. At the same time, external market forces were having an impact on the company: customers were growing tired of interfacing with technical people who were perceived to be (or actually were) incapable of communicating in understandable terms. Also, smaller consultancies, by design, were delivering their services through individuals versed in each of business analysis, project management and technical depth. Agility itself became a key differentiator for certain consultancies, and, the versatility of their delivery people was typically the key enabler behind the scenes.

Versatilist: Coined by Gartner Research; a term to describe people who "are able to apply a depth of skill to a progressively widening scope of situations and experiences, equally at ease with technical issues as with business strategy."

http://en.wikipedia.org/wiki/Versatilist

http://www.gartner.com/press_releases/asset_139314_11.html

The larger IT company endeavoured to change to meet the changing market demands, but had difficulty (at the time) due to the disruption and confusion behind the true aims for such versatility. Organizational change management became the real challenge , in a corporate culture where the technical resources ("techies" as they were called) were often the butt of closed door in-jokes and were the first to have their bonuses cut for achieving certifications (which, ironically, helped to achieve and maintain the company's partner status with a certain vendor). Other examples of coercive and manipulative powerplays were at work, with certain people obviously threatened by such change often acting as an obstacle to change itself. Many technical people, as I did, understood the value of versatility as it was communicated to us, but were unable to see the value in terms of our jobs with the company. One colleague of mine commented: "why would I want to learn more about a business analysis role, when my job specification affords zero bandwidth to perform such tasks?". Another asked "well, I'm one of those business analysts... what does this mean for my job if more of the techies are doing what I am paid to do?"Others still seemed to resent versatility being thrust upon us as yet another "cost-cutting move".

When faced with the prospect of change, we tend to resist; it's human nature, after all. After my first full-time (and somewhat negative) experience with a pure-play BI company, my natural reaction was to "stay technical.... to heck with the business side of things", and after the later experience, I felt that the push for versatility was more in the interests of a given company more than anything else. Nevertheless, change is coming to the IT profession in general, and to people working in the BI space in particular. Versatility IS important -- my experiences have taught me in the long run that, as a BI professional, having an equal balance of skills and knowledge within technology, business and process domains will actually deliver three key benefits:

1. For your customers: They need to know that their system in development to expose financial metrics in useful ways (for example), is in the capable hands of someone who understands BOTH the business case (specifically, the nature and need of such measurements, as well as where they originate in a general ledger) for a financial-based BI system AND the finer details of data profiling, ETL, OLAP and interactive reporting.

2. For your employers: Especially those of you in consulting roles, they need to know that they can comfortably position you in front of a customer, and, can manage customer expectations confidently in project cost-based terms-- you should know how to deliver a solution design which addresses functional requirements, and takes into account the existing environment, user profiles, security and how you will mitigate project risk. You should also know how to position cost-feature tradeoffs for each mid-stream change request during a project AND you should have the leadership skills to motivate, guide and mentor any juniors you are accountable for on a given project.

3. For your career: BI is an increasingly competitive space. With the emergence of the business discipline of Performance Management, broadening your mind to understand methodologies like the Balanced Scorecard is key. Most Performance Management initiatives are cyclical processes, with an aim towards continuous business improvement. A firm grasp of such processes are critical, both towards rounding out your technical and business skills. Regardless of whether you are self-employed, seeking your first job, coasting along nicely in your senior BI architect role, or ready to move to greener pastures in BI, only the most well-rounded professionals will be able achieve and sustain individual competitive advantage.

The reality is that we BI professionals can no longer remain (solely) technical, or, lament that business acumen and project skills aren't in our current job specifications. While a healthy amount of skepticism is encouraged in most companies, relegating versatility to a mere cost-cutting tactic is narrow-minded at best, and in my opinion, ignorant at worst. Take the initiative... if you are classic "techie", then pick up a business book on accounting and finance or operational excellence. If you know about those, then take a closer look at how professional services, manufacturing and retailers/wholesalers differ in their approach to financial planning. Find out more about how valuable intangible metrics are to organizations these days: how about sales and marketing metrics? What about those which pertain to human capital? You get the idea: there should almost always be more to learn in the world of business as a BI professional. While you're at it, get up to speed on Performance Management. Authors like Anthony Politano (Chief Performance Officer), Wayne Eckerson (Performance Dashboards: Measuring, Monitoring and Managing Your Business) and Paul Niven (The Balanced Scorecard Step by Step: Maximizing Performance and Maintaining Results) have provided enough perspective and insight to get you going. Oh, and if you are traditionally a BI analyst , business development manager / account executive, or, only focus on project management, then take the time to learn a little more about the technologies your team or company are delivering. You'll never know when you may be called upon to fill a temporary gap in your company's resource pool.... moreover, you'll have more confidence in demonstrating the products your company uses to deliver BI solutions and, it will give you more credibility in the eyes of your prospects than you may fully appreciate (prospects these days can smell both fear and B.S. in equal measure).

Regardless of you current side of the fence, you don't need to be an expert in these areas you explore outside of your comfort zone. It is simply in the effort you that put in towards being "business intelligent" does the value of being a versatilist show in your marketability and ultimately in your success.

- Adrian Downes