Agile BI – The Future of BI

In a rapidly changing economy, Business Intelligence solutions have to become more agile. This paper attempts to discuss some questions which help in creating an agile BI solution such as: What is Agile? Why agile is so well suited for BI? Which are the key elements that promote an agile BI solution? Also, this paper briefly looks at technologies that can be used for enabling an agile BI solution.


Introduction
Business Intelligence (BI) was defined in different ways.The Data-Warehousing Institute has defined Business Intelligence as "the tools, technologies and processes required to turn data into information and information into knowledge and plans that optimize business actions" [6].Turban has defined BI as "a broad category of applications and techniques for gathering, storing, analyzing and providing access to data to help enterprise user make better business and strategic decisions."[23].The range of capabilities that can be defined as business intelligence is very broad.The spectrum of BI technologies is presented in [16].Most enterprises have hundreds of internal and external data sources such as: databases, e-mail archives, file systems, spreadsheets, digital images, audio files and more.80% of the organizational data are unstructured and semi-structured data.Traditional Business Intelligence systems use a small fraction of all the data available.Also, traditional BI systems use only structured data.The core components of a traditional BI architecture are: ETL tools, an enterprise data warehouse with metadata repository and business analytics (Figure 1).

Fig. 1. A traditional BI architecture
Traditional BI systems use ETL tools for extracting data from multiple sources and temporarily storing those datasets at a staging area.Organizations use data warehouses to aggregate cleaned and structured data.
Business analytics/BI tools include enterprise reporting tools, ad hoc query tools, statistical analysis tools, OLAP tools, spatial-OLAP analysis tools, dashboards, scorecards and advanced analytics.Advanced analytics [20]).

Fig. 3. Waterfall approach to the development of BI solution
But waterfall approach is poorly suited for BI.The main problems of this approach are:  the long times between the system request and the delivery of the BI solution;  users are not involved in the analysis phase, design phase, development phase and testing phase;  it is inflexible to analytical requirements changes;  testing at the end of the development life cycle.
So a different approach is needed to make BI applications more flexible and able to react faster to changing business requirements.In this methodology the BI requirements are divided into small "user stories".An agile BI project consists of a collection of "user stories".Each story is then designed, developed, tested and released to the users.One sprint is a full life cycle of understanding the BI requirements, analysis, design, development and user testing.Each sprint lasts for 1-2 weeks.Users are involved in sprint steps.User stories need to be categorized in one of two ways: product backlog and sprint backlog.Sprint backlog is a list of tasks the team expects to do during the sprint.At the end of each sprint, the business has a deliverable such as a new report or dashboard.Product backlog is a list of all requirements ordered by highest priority of what is needed.The user is responsible for ranking the features on the product backlog.Another concept is the daily scrum that is a short meeting in which every member of the team answers three questions: What did you do yesterday?What will you do until our next meeting?Do you have any problems?Extreme Scoping and Agile Data warehousing are well suited if BI solution includes a data warehouse.Agile Data warehousing is defined as "the application of two agile development approaches -Scrum and Extreme programmingto the specific challenges of data warehousing and BI" [11].Extreme scoping is an agile enterprise data warehousing approach that includes the business integration activities.Also, this approach uses agile principles.The BI solution is separated into multiple releases for iterative development [17].Figure 4 shows a typical agile BI cycle.

Agile Business Analytics (BA)
Besides developing a business intelligence system with agile design methodologies, it's also recommended adopting agile BA [2].Both the design methodology and the tools have to be agile.Agile BA must enable BI users to become less dependent on IT.Also, agile BA must be easier to be used by all types of users.So agile BA should provide at least office suite integration, a business glossary and advanced visual features such as interactive dashboards and drill-down capabilities.A recent study by Forrester showed that agile BA should be integrated In memory spreadsheet -spreadsheet loaded into memory  -no modeling required -access by third-party tools.

Microsoft PowerPivot
In memory "associative" data model -loads and store all data in an "associative" data model (array) that runs in memory.
-all joins and calculations are made in real time. -less modeling required than an OLAP based solution  -limited by physical memory  -some scripting required to load the data QlikView

Agile Information Infrastructure
True agility is reached by making all parts of a BI system agile.An agile BI solution can be seen as consisting of two layers: an agile information infrastructure layer and an agile analytic layer (figure 5).Information infrastructure addresses how the data architecture and data integration infrastructure ensure agility to react to changing business requirements.An agile information infrastructure must be able to extract and combine data from any data sources, internal and external sources including relational, semi-structured XML, multidimensional and "Big Data".How to get an agile information infrastructure?By using data virtualization.Data virtualization is "the process of offering data consumers a data access interface that hides the technical aspects of data stores, such as location, storage structure, API, access language, and storage technology" [12].According to [13], [14] data virtualization "is the technology that offers data consumers a unified, abstracted, and encapsulated view for querying and manipulating data stored in a heterogeneous set of data stores.Data virtualization means on-demand data transformation, on-demand data integration, and on-demand data cleansing".Data virtualization can be implemented in many ways such as: using a data virtualization server or placing data sources in the cloud.Traditional BI systems use ETL tools for extracting data from multiple sources and temporarily storing those datasets at a staging area.As opposed to ETL tools, data virtualization server:  allows the source data to remain in their original locations;  eliminates staging of the data;  abstracts source data, resolving structural and semantic issues;  generates business views and/or data services that provide data required.A business view is conceptually equivalent to a relational view.The views can read data from multiple data sources including: relational databases, multidimensional databases, text files, XML documents, spreadsheets, HTML pages, NoSQL databases, and so on.Applications access source data through the business views/data services using different interfaces such as: JDBC with SQL, ODBC with SQL, SOAP/XML and MDX.Data virtualization server offers:  data modeling capabilities;  data profiling capabilities;  data transformation capabilities;  on-demand data integration capabilities which result in more agile BI systems.

Fig. 5. A high level architecture of an agile BI system
Many data virtualization servers are currently available such as: Composite Information Server, Denodo Platform, IBM InfoSphere Federation Server, Informatica Data Services, and so on.A high-level architecture of an agile BI system is illustrated in Figure 5.This architecture is based on data virtualization server.Data virtualization server examines the data source structure and the resulting metadata is stored in metadata repository.Then you can create business views or data services using the data source's metadata In conclusion, an agile BI solution requires: an agile development methodology, agile BA and an agile information infrastructure (Figure 6).Also, Figure 6 briefly summarize strengths of an agile BI solution.The Table 5 presents a comparative analysis between traditional BI solution and agile BI solution using the following criteria: business requirements, integration approach, data timeline, data refresh, information delivery, data source format, development methodology, development cycle and BA.

Conclusion
In conclusion, the main reasons for implementing agile BI are:  constantly changing business requirements;  inability of IT to meet business user demands;  slow access to information.
Agile BI solutions enable organizations to adapt to changing market conditions.This paper has identified the key elements that together promote an agile BI solution.There are plenty of technologies that can make an agile BI.This paper briefly looked at technologies that can be used for enabling an agile BI solution such as: in-memory as: Scrum, Extreme Programming, Crystal, Dynamic Systems Development, Lean and others.Many of agile software development characteristics can be applied to BI projects from team structure, project management, BI system design, BI system development and analytical techniques [5].The most popular agile development methodologies for BI projects are: Scrum, Extreme Scoping and Agile Data Warehousing.The main concepts of Scrum [21] are: user story, product backlog, sprint backlog, sprint and daily scrum.

Fig. 6 .
Fig. 6.The key components that promote an agile BI solution

Fig. 7 .
Fig. 7.A traditional BI workflow versus an agile BI workflow

Table 3 .
A SWOT analysis for implementation of in-memory BI solutions

Table 5 .
Traditional BI versus agile BI /dimensions.Then the data warehouse modeler decides how the dimensions and facts should be integrated into data warehouse.Also, he develops data models for staging area, warehouse database and cubes.The ETL developer develops ETL code to load data.Then, the BI developer develops cubes and dashboards.The Data quality analyst verifies the quality of dashboards.Then dashboards are published to server and the customer uses these dashboards.The BI consultant advises customers in the fields of information management and in selecting the most suitable BI solution.He is the first point of contact for the customer.He has experience with modern business intelligence techniques, data modeling, ETL tools, software development cycles, and so on.In short a BI consultant is responsible for the requirements gathering, the design and the development of BI solutions.