How to Get the Most Out of a Product Evaluation
A successful evaluation starts with asking the right questions. Protip: “What makes [insert software application here] the best?” isn’t one of them.
Just as there’s no such thing as “the best bicycle” or “the best browser,” there’s no such thing as the best BI solution. And yet, prospects ask us this question all the time, so often that one of our sales representatives asked if I could help draft a scripted answer. I told her it was impossible, that it’d be doing them a disservice not to reframe the question.
It’s not why we’re the best. It’s for whom.
To properly evaluate anything — a app, a film, an electric toothbrush — you need some kind of objective measure or standard. Without that, you’re comparing apples to oranges. If one toothbrush has bluetooth connectivity and the other has sonic action, the only way to choose between them is to determine which features will work best for you. Context matters.
So the way to get the most out of a software evaluation is to define a successful proof of concept (POC). A POC is a context-specific means of proving or disproving the feasibility of a method or theory by applying it in a project. In an embedded BI context, planning a POC and qualifying its success requires intimate knowledge of your company’s needs and where they are likely to intersect with the product under review. Because a POC of this magnitude would ideally test not only the platform’s technological feasibility but also its fiscal and experiential feasibility, formulating one can be a daunting task.
Thankfully, there are some tips and tricks you can use to get your team closer to conceiving of a POC. The great thing is, once you’ve designed one, you can reuse it again and again to evaluate other similar platforms.
Step 1: Build a technical features list.
This list is the meat-and-potatoes of a POC and dictate what features you will attempt to implement during your evaluation. Here are some methods for developing that list:
Identify what you like/dislike about your current BI tool or practice.
- Develop a list of technical specifications. (For example, what operating system(s) you use, what browsers your clients need supported, what databases you’ll need to access, etc.)
- Conduct a competitive analysis. Determine what BI features your competitors offer and what their clients say about them. Determine whether those features are also likely to meet your clients’ needs and add them to the list if so.
- Develop user personas or talk to users themselves. The goal is to find out what an end user is likely to need in terms of workflow. If you can’t interview users about their operational reporting needs or observe them in their work, build plausible personas for an array of user types (the power user, the novice user, the manager, etc.) and use them to anticipate the features they’d find useful.
Step 2: Prioritize your technical features list.
It’s okay for this to be long and exhaustive, but make sure to prioritize it. One method is to segment the list into “Must-Haves,” “Good-to-Haves,” and “Could-Haves.” You could also assign each list item a weight between 1 and 5, determine which the application does and does not support, then add up the points for each category. Or, you could combine these methods and only weight the items not in the “Must-Have” category.
Step 3: Add business features to the list.
It’s of course important to consider more than just the product specs, as purchasing third-party software is, in many ways, akin to forming a business partnership. In a separate section of your list, develop a system for evaluating the non-technical aspects of the transaction, including:
- Payment plan. How does the cost of the software compare with your budget? How will cost change over time and as you grow? As your clients grow?
- Support. Keep notes on the quality of the company’s support team over the course of your evaluation. Is it prompt, clear, and effective? Read client reviews to determine whether you can expect the same level of service after signing.
- Services. Does the contract come with an allotment of on-site service hours? Are these available at additional cost? What do existing clients have to say about these services?
- Special Builds. In the event that your company needs a rush on a build in order to meet a deadline or satisfy a client, is there a contractual provision for special builds? Does the application have extensibility features that will mitigate your need for special builds?
- Training. Is the software documentation something you can either adapt or pass on to your end users? Does the company offer any training programs that would help users become more proficient at using the application?
- Security. If you have security requirements with which you need to comply in order to serve your clientele, will the company agree to have their code meet the same standard?
Step 4: Plan Your POC project.
Think of your features list as a pantry full of ingredients and your POC project as a cake. Your goal is to use the maximum number of ingredients in the making of this cake because the cake is the test to determine whether the ingredients do what they should. Be sure to:
- Determine which reports, dashboards, schedules, etc. you’d like to build. Select reports that range in difficulty and complexity so that you can get a sense of how different users will experience the product.
- Determine which features you’d like to configure or customize. Think about what product elements you’ll want to expose/hide for different user classes and experiment with those settings. This might include things like CSS styling or language translation.
- Decide who you’d like to interact with the application as part of the project. Consider having individuals from outside the evaluation team interact with the application in front of an observer. The more diverse this population, the better it will help your team evaluate its usability.
- Define how much time you’re willing to allot to the project before you begin. This is to minimize your chances of reaching a point of diminishing returns. Some evaluations have a defined endpoint, but for those that do not, it can save your team time and money to enforce one internally.
Step 5: Define a successful POC.
If you’re using a weighted system, how many points does the BI solution have to earn to qualify as successful? Remember to factor in your notes on business features as well.
Step 6: Evaluate.
Bring your team together to review the results of your POC and determine whether or not it met your success qualifiers. Don’t be surprised if there’s some disagreement and debate in the process: no matter how objective you strive to make your rating system, there will necessarily be room for interpretation. Once you’ve evaluated a handful of solutions (somewhere between 3 and 5 is the usual recommendation), compare their scores and try to reach a consensus (or, if not consensus, at least a decision) about which is best for you based on your rubric.