Integration Trends for 2006

IT organizations devote a significant percentage of their budgets to addressing integration issues and evolving their enterprise-wide integration infrastructure. Investment in integration technology and products will continue to increase this year. Because of this, I thought it would be interesting to explore where this money is likely to be spent in 2006 in this year's first article.

From an integration perspective there are five areas that should be considered. The first four are associated with the main types of integration technology: user interaction, business process, applications and data. The fifth is concerned with how to bring these four types of integration together into an enterprise integration architecture. Let’s examine the likely trends in each of these areas for 2006.

User Interaction Integration

The direction over the past few years has been toward using a Web interface to support user interaction with corporate applications and data. The business portal is the favored technology for supporting integration at the Web-based user interaction level.   

Portal products and technologies have changed dramatically since their introduction over ten years ago. Early portals were developed using products from independent software vendors that focused specifically on portal solutions. Almost all of these vendors have either gone out of business or been acquired. Content management companies, such as Vignette, primarily own the ones that remain.

Today, the big infrastructure vendors like BEA, IBM, Microsoft, Oracle, SAP and Sun dominate the portal marketplace. Most of the portal interfaces shipped with packaged applications also utilize the technology from the infrastructure vendors.

The focus in 2006 will be on how to create a single business user collaborative workspace by merging Web-based interfaces with the Microsoft Office desktop environment. At the workgroup level, Microsoft’s increasing integration of Office with SharePoint will lead to workgroup environments dominated by the SharePoint portal. The challenge will be in integrating this Microsoft.net workgroup environment with enterprise-level portals, which are based on Java-driven products from the other big infrastructure vendors like IBM and Oracle. Microsoft’s relationship with SAP is also likely to have an impact in this area.  

Business Process Integration and Application Integration

In the past, business process integration and application integration could be viewed as separate technologies, but the dividing line between them has become so fuzzy that they can now be considered a single technology. The merging of business process and application technology has been driven largely by both vendors and IT organizations, who are moving toward a service-oriented architecture (SOA) based primarily on Web services.

The trend toward a SOA and Web services will continue in 2006. This environment will not only be used for integrating operational business processes, but also for business intelligence processes. This year will see significant growth in the use of operational business intelligence. In some projects, operational business intelligence services are likely to be embedded in operational processes for in-line analysis where fast action times are required. Examples include detecting fraud, processing insurance claims, issuing credit cards and loans, tracking potential financial issues, etc.

Data

More and more organizations are now viewing data integration as an enterprise level problem, not just an issue to be solved when building data warehousing and business intelligence applications. A recent TDWI study showed that 18 percent of companies have developed an enterprise level architecture for supporting data integration in their organizations.

As companies move toward an enterprise approach to data integration, they must reuse and share data integration expertise and resources. This can be achieved by creating a data integration competency center. This year will see a greater use of the data competency center concept and on building an enterprise-wide data integration architecture.

As I mentioned earlier, one of the biggest growth areas in business intelligence lies in its use for operational processing. Operational business intelligence typically requires access to more current data than the data provided by a traditional data warehousing system. As a result, interest in creating low-latency data stores for operational reporting analysis is increasing. A recent Business Intelligence Network Research Study on right-time data integration discusses this topic in detail. This report is highly recommended for anyone interested on this subject.

Another important growth area is master data management (MDM), which causes much debate from a terminology perspective. Master data is reference data that defines and supports the key business objects that underpin the main business processes of a company. Examples of this include customers, employees, finances, products, brands, suppliers and partners.

There is more to master data management than data integration. Master data management solutions usually offer collaborative, business intelligence and workflow capabilities. They also help with the solving of semantic data issues across applications. It is the ability to address specific business area issues that provides the major benefits of most master data management solutions. This is why many MDM vendors are now marketing vertical business area master data management offerings. Claudia Imhoff and I are working with the Business Intelligence Network on an master data management research study to be published later this year. So keep an eye on this website. 

Environment

The last area to discuss is how to create a complete business integration environment based on the technologies outlined above. The major issue that most large companies will face is that their enterprise-level systems will be based on Java and software from the big infrastructure players, though most of their workgroup environments will be supported on a Microsoft platform. This will probably be the number one integration issue for companies this year, especially since the Microsoft world is far less open than the enterprise-based Java one. While the Linux and open source enthusiasts will say this problem can be solved by replacing Windows with Linux and OpenOffice, this is not as easy as it sounds, even though some companies are doing this.

The common denominator between the Microsoft.net world and the enterprise Java world is Web services. This is why companies are taking SOA and Web services so seriously. Web services are one of the main ways of connecting the workgroup world to the enterprise. It will be interesting to review the situation again in January 2007. We will be able to see how far organizations have moved in this direction, as well as how effectively the vendors can make this integration easier to do than it is today.

Improving Business Intelligence Usability

Until recently, business intelligence product vendors have focused mostly on improving their reporting and analytical processing. Although this increased functionality has benefited power users, such as business analysts and information workers, it has not effectively encouraged the use of business intelligence by less experienced users. Common examples of this are business executives and line-of-business (LOB) managers.

The tide is now turning, however, as vendors seek to expand the sales of their solutions to a broader user base. There is now more awareness by business intelligence vendors on the need to improve the usability of their products. They are now listening to customer complaints about the need to reduce IT workloads. Customers feel this will enable business to become more self sufficient through self-service applications. They are beginning to realize that Microsoft Office and Excel are successful, not only because of Microsoft’s dominance, but also because less experienced users find these products easier to use than other business intelligence tools.

The movement toward improving usability is evident from several of the major business intelligence vendors as evidenced by their upcoming product releases. There is now improved integration with Microsoft Office, increased support for business portals, and a focus on today’s favorite buzzword—the business dashboard. Given this trend, it is important to understand the relationship between business dashboards, business portals and collaborative products like Microsoft Office.

The best approach to examining usability is to start with business user requirements. Today, business users must have a personalized workspace (see Figure 1) for a collaborative environment. This workspace will give them the ability to access all of the business content needed to perform their daily tasks. The business content may consist of business processes and workflows, business applications and/or business information. In turn, the business information may come from operational systems, business intelligence applications, content management systems or a collaborative environment. This business workspace is provided by a business portal. 

Figure 1: The Business User Workspace

A business user workspace enables users to perform four main tasks: run, search, navigate and collaborate. From a business intelligence viewpoint this could involve:

• Running a stored business intelligence object (query, report, analysis or model), business intelligence tool or application.
• Searching for business intelligence objects (queries, analyses, reports, templates, models, results, metadata, etc.) and business information.
• Navigating a business intelligence (metadata) object tree; drilling up, down or across business intelligence results; or following web links to related information or applications.
• Collaborating with other users or groups via e-mail, instant messaging, live meetings, etc.; or sharing information with other users or groups. 

Business intelligence is delivered to a business user workspace using a business dashboard. A business dashboard provides access to the services of underlying business intelligence tools and applications. The objective of a dashboard is to hide the complexity of the business intelligence environment. The relationship between a business dashboard and a business portal.   

Figure 2: Business User Workspace Hierarchy

A business portal provides access to business content (using the run and navigate tasks) and offers search and collaboration capabilities. Similarly, a business dashboard provides access to the business intelligence environment (also using the run and navigate tasks) and sometimes related business information. Searching and collaboration should be done using the business portal, rather than the business dashboard, unless the search and collaboration services can be shared by both the portal and dashboard environments. One issue here is that some business intelligence vendors now offer their own collaborative tools, which are specific to the business intelligence environment. I believe this is the wrong approach to collaboration. 

The new Microsoft Business Scorecard Manager 2005 is one of the best demonstrations of an integrated business user workspace and business intelligence environment. This product not only provides a powerful and competitive business intelligence scorecard and dashboard environment, but also tightly integrates with the Microsoft Office and Microsoft SharePoint portal environments. Products supported by this include Excel, Visio, Project, InfoPath, SharePoint, MapPoint, Access and Analysis Services.    

(Aside: Many people confuse scorecards with dashboards. A scorecard is a set of business intelligence results that compares metrics of actual business performance to business goals. Conversely, a dashboard is a visualization object that is used to display scorecard results.)

In summary, improving the usability of business intelligence solutions involves integrating business intelligence dashboards with business portals, while adding a collaborative environment. As you evaluate new business intelligence vendors’ offerings, one should determine how well their new usability capabilities can support a personalized business user workspace

Business Integration: Enterprise Portal Evolution

I’ve often stated in business integration articles that the business portal is becoming the standard approach for providing users with a role-based and collaborative workspace for accessing, sharing and communicating about business content. While this is still true, the evolution of an organizations’ portal technology has made it increasingly difficult to support a single enterprise portal. These portals could provide everything that each user in the organization requires.  

To understand the current role of a business portal, we need to examine how portals are being deployed and used by organizations. We must also consider how portal technology is being packaged and delivered by vendors.

The role of business portals is to give users access to the business services and content needed for their jobs. These services and content are managed and provided by five main types of applications, as well as their underlying data stores (see Figure 1):

1. Collaborative applications that provide e-mail, instant messaging, discussion groups and bulletin boards, web conferencing and online presentations, online white boards and screen sharing and remote learning.
2. Workgroup applications that support search, document sharing and workflows, the management of workgroup documents, web pages, digital media and local offline data.
3. Enterprise content management applications that provide similar capabilities to workgroup applications, but at the enterprise level. These applications also support large volumes of operational documents and are able to perform records management.
4. Business intelligence applications that report on, analyze and deliver business information about strategic, tactical and daily business initiatives and operations.
5. Business transaction applications that run day-to-day business operations.

Figure 1. The Business Portal Environment

The services and content delivered through business portals may be used by employees, customers, suppliers and partners. Employees may access these services and content at the workgroup or enterprise level. The access level used will depend partially on the business tasks being performed. Users that simply want to retrieve business content, run business transaction applications and do simple collaborative tasks can operate quite successfully using a web-based enterprise portal. In most cases, this thin-client interface is suitable for customers, suppliers as well as partners.  

Employees who need more intensive business content at work are more likely to operate at the workgroup level. These same employees also want to use a wide range of business intelligence and collaborative capabilities. In this situation, a web-based interface may not be sufficient because many workgroup tools employ a rich client desktop interface. An example of this is Microsoft Office.

A web-based, enterprise-level business portal will meet the needs of most external users and employees wanting to access information. In these cases, the portal products given by infrastructure vendors like BEA, IBM, Oracle, etc. will match these needs. However, information workers will need a workgroup portal environment that supports both thin and rich clients. This workgroup environment is dominated by Microsoft, which is tightly integrating Microsoft Office with Microsoft SharePoint. This integration is likely to continue in the next releases of both products.

The major issue facing many organizations is whether to employ Microsoft SharePoint at the enterprise level. An alternative is building a bridge between the Microsoft Office/SharePoint workgroup environment and an enterprise portal from a different vendor. The answer to this dilemma will depend somewhat on the size of the company and its IT strategy. Businesses using the Microsoft operating environment can often successfully use Microsoft SharePoint at both the workgroup and enterprise levels. Large corporations, who use both Java-based application development platforms from infrastructure vendors and Microsoft workgroup users, will need to build a bridge between the two environments.

Companies using portals provided by applications vendors, such as SAP, may need to build connections to Microsoft workgroups too. The joint SAP and Microsoft Mendocino project is really the first step to building a pathway between the Microsoft and SAP environments. IBM also recognizes the distinction between the workgroup and enterprise requirements, with respect to business portal usage. Here, IBM’s strategy is to provide the WebSphere Portal at the enterprise level, and to link this portal to the IBM Workplace product. This will be done for both thin and rich-client (IBM calls these managed clients) workgroup computing.

In summary, organizations can deploy web-based enterprise portals to provide employees and external users with personalized business user workspaces. Information workers using powerful collaborative, content management and business intelligence capabilities will also need a thin and rich-client based workgroup environment. Such an environment can interoperate with the enterprise portal. 

The Smart Business Intelligence Framework

I’ve often made the point in my articles that business intelligence is no longer just nice to have, but is essential to business success. I’ve also commented at the same time that business intelligence applications and their underlying data warehouses can only support the needs of the business if they are tightly integrated into the overall IT environment. To highlight the importance of business intelligence and the need to integrate it into the enterprise, I developed the concept of the Smart BI Framework. The latest version of this framework is shown in Figure 1.  

Figure 1. The Smart BI Framework

Copyright BI Research and Intelligent Solutions, 2005.

The Smart BI Framework brings together the four forces that drive business operations and the IT systems that support them. These four forces are people, plans, processes and performance.

A company’s people are the underlying foundation on which the business is built. Without good employees a company will fail. How people perform their role in the organization is changing. The speed of business today means that people can no longer sit in ivory towers, or control and restrict the flow of information within the organization. If information is power then it must be made available to the people that need it for their jobs.

Key to collaboration and the sharing of information is knowledge management (KM), which brings together portals, content management and collaboration tools. The growing importance of business intelligence also means that it too must be integrated into the KM environment.

As senior executives define business plans and goals they must communicate them down through the corporate hierarchy. Targets must be developed and measured, and employees must be told what is expected of them. Employee compensation should generally be tied to achieving expected targets. Planning, budgeting and forecasting systems form the basis of the planning process, but collaboration capabilities are required for communicating plans and goals, and business intelligence is essential for monitoring and managing targets. Methodologies like balanced scorecards are also valuable for formalizing the planning process and managing targets.

Once business plans and initiatives are agreed on, they are implemented in business processes. Business process management is a growing technology for modeling, simulating, deploying, integrating and monitoring business processes. At present, process management is used primarily with operational business transaction applications, but the need to manage document and information workflows is bringing process management concepts and technologies into the collaborative application environment.

Business transaction applications run business operations and associated business processes and underlying activities. The role of business intelligence applications is to monitor, analyze and report on those operations. The output from business intelligence applications is used to determine how well actual business operations are doing, compared against business goals and targets. If these business goals and targets are not being achieved, then either business plans or business operations must be adjusted accordingly. This aspect of business intelligence is often called business performance management, which is easily confused with business process management, especially given that process management also supports the monitoring of business performance.

Business performance management is a term that is becoming increasingly abused by vendors. Vendors will use the term to describe a product even if it simply creates a business dashboard showing basic performance measures that are unrelated to business plans, goals or targets. A true business performance management application is closely tied to business plans and planning systems so that performance measures can be related to business goals and targets.          

Most business performance management applications deliver information that is reactive in nature, i.e., the information produced identifies business problems after they have occurred. Ideally, business users would like to be able to predict or anticipate business issues before they occur. The integration of business intelligence predictive technologies and planning methodologies into the business performance management environment helps satisfy this requirement.       

At present, business intelligence is data-centric, but as it becomes more integrated with business operations it will need to become more process-centric so that business intelligence results can be more easily related to business processes and their associated business activities. This involves integrating performance management and process management technologies. Perhaps the term to use here is business process and performance management, or BPPM. This term would at least remove the current industry confusion over the BPM acronym!

BPPM would allow business intelligence to be integrated into business transaction processes and also allow business processes to be added to business intelligence applications. An application example in this latter case would be for a performance management application to alert a business user about a business problem and provide a guided analysis workflow or procedure that helps the user investigate the problem in more detail based on best practices.

At the heart of a business intelligence system are the operational data store, enterprise data warehouse and data marts that supply the integrated, clean and consistent data for analysis. Many traditional data warehouse implementations have been deployed using the Corporate Information Factory architecture developed by Claudia Imhoff of Intelligent Solutions. As business intelligence becomes more integrated into the business environment, this traditional architecture must evolve to support the technologies and techniques outlined in this article. I have been working with Claudia to design an Extended Corporate Information Factory that supports the Smart Business Framework outlined in Figure 1. The Business Intelligence Network will publish an article on the Extended Corporate Information Factory soon.

We can see then that a Smart BI Framework involves connecting together business intelligence, business transaction and collaborative applications and their underlying data and information stores. Further enhancing are integration connections to business planning systems, and support for knowledge management, business process and business performance management technologies. Such a framework brings together the four main business drivers of an organization: people, plans, processes and performance.