Industry Analyst John Myers on Finding Scalable BI
John L Myers is Managing Research Director of Business Intelligence and Data Warehousing at Enterprise Management Associates (EMA), a leading industry analyst and consulting firm headquartered in Boulder, Colorado. John delivers comprehensive coverage of the business intelligence and data warehouse industry with a focus on database management, data integration, data visualization, and process management solutions. His work is informed by over a decade of experience with business analytics implementations associated with the telecommunications industry. John’s passions are rooted in analytics, business process, and the what it takes to inform diverse business models.
We got in touch with John to learn more about his strategy for finding scalable BI platforms.
What would you say is the top mistake organizations commonly make in purchasing embedded BI?
Their number-one mistake is going for something quick and easy. An embedded solution that only meets today's limited functional requirement is a solution that often needs to be replaced as an organization matures its embedded strategy. Looking down the road helps to make embedded successful today and in twelve months.
I think the reason so many companies fall prey to this mistake is that they don’t always understand the potential loads associated with embedded BI. Not understanding this can be almost as bad as selecting an embedded solution that won't meet future functional requirements. Embedded solutions need to handle user traffic and query loads if the application that the embedded analytics “lives” in attains "hockey stick" or exponential increases in adoption.
What questions should decision makers and architects ask embedded BI vendors in order to ascertain the product’s scaling potential?
The first thing I would do is ask about load testing metrics. Be sure to also ask how the solution is designed to scale, whether vertically with scale-up or horizontally with scale-out (preferable). If you are going to scale up, get information on the types of transactions per core that can be achieved. If you plan to scale out, look for details on linear scalability by the addition of each additional resource unit (server, blade, etc.).
Second, get a reference customer call set up, preferably at the company staging the product’s largest implementation. Request to speak to that particular client (usually under an NDA), and you will determine if the vendor has a real implementation. Note: most vendors will only set up a reference with happy customers, just like a job reference. During the discussion, however, you’ll often get a real-world appraisal of what the solution does or does not do well.
What red flags should companies look out for in evaluating the scalability of the license agreement?
If a company cannot provide you with the load testing metrics and reference I mentioned before, that’s a very obvious red flag. If they don’t have the reference, it means they don’t have a satisfied customer with a large-scale implementation; and if they don't have the scaling chart, it either means they haven't load tested their system or the load test revealed issues.
Another red flag is if the vendor doesn’t talk about concurrency. Scalability isn't just about the number of transactions taking place over a given period of time—it's also about the number of transactions happening simultaneously. This is often referred to as "concurrency," and that concurrency level is another thing that vendors should have either in their documentation or as part of their pitch. The bulk number of transactions is one thing, but if I have ten thousand people hitting the system or ten thousand groups all hitting it at the same time, I need to know what my response rate going to be.
You can follow John Myers on Twitter at @JohnLMyers44 and read his articles on EMA’s blog. For more guidance on the BI purchasing process, check out Getting the Most Out of a Product Evaluation or download our Proof-of-Concept (POC) Checklist.