BI and the Cloud

Wayne Eckerson from The Data Warehouse Institute has an interesting post about Implementing BI in the Cloud. He mentions that BI in the Cloud faces four constraints:

1) Customization or application fit

2) Ongoing cost of transferring data to the Cloud

3) Data Security

4) Vendor viability

Wayne wraps up his post with the following conclusion:

BI for SaaS offers a lot of promise to reduce costs and speed deployment but only for companies whose requirements are suitable to cloud-based computing. Today, these are companies that have limited or no available IT resources, little capital to spend on building compute-based or software capabilities inhouse, and whose BI applications don’t require significant, continuous transfers of data from source systems to the cloud.

I tend to agree with the following high level thoughts:

1) BI on the Cloud is not for everybody (yet)

2) Due diligence is necessary to reduce risks on data security and vendor viability

But Wayne's post raised several questions in my mind:

Integration. There are several ways to customize an application. For example, a multi-tenant architecture like Salesforce.com offers endless possibilities to customize and extend every single instance. Are these customizations unprofitable? No they are not, they are part of the application and they do not require changes to the underlying code. Can the same level of customization apply to BI? Absolutely. Wayne mentions briefly Platform as a Service but his focus is towards custom application development (although his chart shows "DW as a Service"). An intriguing approach to offer a BI platform as a service would be to setup something like MicroStrategy and configure it in a multi-tenant fashion. The underlying layer of IaaS would support the data repository while the top layer of SaaS could support ad-hoc reporting, vertical applications or full customizations on top of their API. Would this be unprofitable? Not at all. Wayne makes another good point regarding integration:

So, unless the SaaS vendor supports a broad range of integrated functional applications, it’s hard to justify purchasing any SaaS application.

But from my experience, successful enterprise wide deployments need to focus on integrating subject areas at the data level. This is an architecture and design challenge. A well integrated data repository will support integrated functional applications seamlessly. It is about the underlying data not only the application.

Ongoing Data Transfers Costs. Is this really a significant constraint? How much data does the typical Data Warehouse has to incorporate on a daily basis? The cost to transfer data to Google's App Engine is $0.10 per GB. Moving a TB a day would cost around $3,000 per month (I'm not suggesting using BigTable as a DWH repository yet). As Data Warehouse costs go, this does not seem unreasonable. Amazon is running a promotion right now that would bring that cost down to $1,000; hardly a deal braker. Latency and complexity can complicate this data transfer. This is to be expected because 99% of them were not designed with the Cloud in mind. Which brings me to my final point.

I mentioned using MicroStrategy as a BI platform on the cloud as an example to make a point. I believe that successful Cloud applications need to do more than just cloning their on-premise counterparts. They need to leverage the Cloud inherent qualities, for example elastic computing power. The nature of the Cloud can enable ongoing ETL: receive a copy of the transaction on the fly via a web hook, cleanse, transform and aggregate in real time or a few times a day at least. How about Map Reduce? I think this technique will allow to create more powerful analysis over more data, faster and easier.

Rigid applications built with yesterday's patterns will struggle to survive, in the Cloud or elsewhere. The Cloud is an open environment by definition, its openness will facilitate the integration of multiple data sources from inside and outside the corporate firewall. This integration will support a next generation of cross-functional applications. Bandwidth and storage costs continue to drop very rapidly and will cease to be a major consideration in the near future. New design principles (e.g. scale out vs. scale up) will enable more sophisticated analysis over ever larger datasets (Google analyzes over a PetaByte of data every day). With over $1B in sales Salesforce.com is the most successful SaaS provider. They host more than 55k customers, well over 1M users and every day execute more than 30M lines of customer code. If they can do it, I'm convinced the next BI leader in the Cloud will do it as well. That is how I see it.

Federated Applications

Intuit recently launched a new ambitious program to deliver Federated Applications on their platform. These applications will be available to Intuit customers via a "marketplace" (my term not Intuit's). Intuit has a number of SDKs to use their Cloud platform (Flex seems to be a strategic platform for them) but Federated Applications will not be constrained to a specific technology because they will be able to use a REST API. From the distance this program looks similar to Salesforce's successful AppExchange--approaching 1,000 published applications rapidly--but after a closer look it is important to highlight two intriguing differences:

1) Intuit takes care of the billing. It collects the monthly fees, pays the application developer and keeps a %. To many developers this is a nice service, they just know that every month money will be deposited in their bank accounts and they don't have to worry about paying Intuit. Intuit gets paid right away.

2) Federated applications have the ability to read data from Intuit applications but ALSO from any other Federated application in use by the current user. To make this happen Intuit requires that all applications meet certain integration requirements. While this is an extra step for most application developers the amount of work does not appear to be excessive (i.e. weeks not months). These requirements help to deliver a more consistent user experience and they make it easier for customers to try new applications. Another interesting consequence is the potential Network effect among applications. Federated applications are expected to publish their data objects to an open and shared environment. This means that application A can read/write data from Intuit applications but also from federated application B. This data exchange can increase the value of individual applications significantly (the more connected the more valuable). Of course, each application would need to know what type of objects to expect and what to do with them but given a common framework it should not be too difficult.

Compared to other Cloud providers (e.g. Salesforce) Intuit would seem to be off to a slow start but I find their vision intriguing and ambitious. More importantly if Intuit succeeds in recruiting useful applications this strategy could become very profitable for all involved.

The Cloud is Crossing the Chasm

Cloud Computing is crossing the chasm. Every other week you hear about the deployment of Cloud Computing applications to thousands of users by leading corporations and governments. These deployments go hand in hand with the announcement of new Cloud offerings by leading technology vendors and service providers. You can still find disagreements about the size of the market or the actual definition but there is no doubt that Cloud Computing is having a great year and the best is yet to come. These are some of the many news/announcements that caught my eye:

1) This is a couple of months old but I have to mention Oracle's acquisition of Sun. It was interesting to see Oracle make the move after all the talk about IBM buying Sun. But the implications to the industry are far more intriguing. How is this acquisition going to impact HP? Will IBM see any impact from Oracle 'owning' Java? Will ORCL be able to manage Hardware and Software effectively? Oracle has been a bit slow to announce a compelling Cloud strategy. Sun made a late announcement about their software strategy and more recently another one about consulting services for the Cloud. This is an intriguing combination, it might seem like the odds are against them but you should never underestimate Oracle.

2) TIBCO launches TIBCO Silver an application development platform for AMZN's AWS. I'm intrigued by this announcement. Although some have labeled it "Amazon for Dummies" I'm curious to see if TIBCO will be able to leverage some of its sophisticated analytical technology for intelligent scaling and SLAs.

3) In close partnership with RedHat (RHT) Verizon (VZ) launched a new service offering called Computing as a Service (CaaS) "With CaaS, you have access to bandwidth, servers, storage, and firewalls with dynamic real-time control over what, when, and how those resources are deployed". This service will put pressure on traditional/smaller hosting providers looking to transition to the Cloud. It will also help enterprise customers feel more comfortable about embracing the Cloud.

4) The brilliant Ray Ozzie elaborates on Microsoft's view of the Cloud. You know it is serious when Microsoft gets serious about it. It is hard not to imagine a future with public and private clouds, unless of course, networking technology evolves to a point where the difference is irrelevant. Given MSFT's impressive footprint in the enterprise (Exchange, Sharepoint, SQL Server, etc.) they seem to be in a perfect position to dominate that hybrid world with Azure. If you can offer greater scalability, lower costs and greater flexibility leveraging the same skills and the technology you already have then you have a winning recipe. The question is "Will they be able to execute?"

5) CSC to offer cloud services. Now if it wasn't enough that Microsoft is dead serious about Cloud Computing, CSC one of the largest and most successful government contractors will leverage its security and strategy expertise to offer cloud services. I'm expecting Lockheed, EDS and others to follow with similar offerings. These services will certainly lower the risk for CIOs in the public sector.

6) More fuzzy math, this time courtesy of Gartner in one of their estimates for the size of the Cloud Computing industry. Inconsistent estimates are the consequence of inconsistent market definitions, I can see that. I just hope that there are no hidden agendas.

7) Google Apps is profitable and growing. Although not a direct replacement of MS Office, Google Apps continues to capture market share one enterprise customer at a time. Although mostly driven by cost savings in Email you can expect the adoption of Google Docs to increase as new and improved versions continue to roll out over the next year (including of course my favorites: App Engine and SDC).

With so much activity it is hard to believe that the best is yet to come but trust me, it is. Hang on tight.