At Capterra, we talk to software buyers every day. And, in our experience, most of them are looking for answers to the same three questions:
- What is everyone else using?
- Which system is the easiest to use?
- Which solution is the most cost-effective?
We’ve been answering the first question with our Top 20 Most Popular Software Reports for five years now. Across dozens of industries, we’ve been ranking business software by the largest number of users, customers, reviews, and social following to help buyers understand the major players in each market. But while focusing on what everyone else uses is a great starting point, it only solves part of the puzzle for buyers.
In order to address buyers’ two additional questions, Capterra has decided to create two new reports. The Top 20 Most User-Friendly Software Report and The Top 20 Most Affordable Software Report aim to provide more insight into software functionality and price, which have been overlooked in our popularity evaluations.
In short, we hope to help fulfill Capterra’s mission of being ridiculously helpful by providing an easily digestible report that assists buyers by answering their central questions from different angles.
How does Capterra evaluate software solutions for these reports?
Our evaluation process starts with independent research. We try to identify the needs of the typical user for whatever software space we’re evaluating. We then identify major, promising, and rising vendors in that space.
In this initial stage, we work directly with the software vendors themselves to provide information about features and pricing, allow access to the software portals, and direct us to the appropriate training materials.
The foundation of our Most Affordable report rests on determining what buyers will get from a software solution and how much, for what they’re getting, it will cost. To find the answers to these questions, we first create a “feature set,” which includes features we’ve determined that buyers expect to find in a particular type of system. We score systems based on whether or not they offer those core features. We then create a pricing scenario, which asks how much it would cost for a typical buyer to get as many of those features as possible. This number determines the “total cost of ownership” of a solution. The pricing scenario also includes a certain level of support and training that buyers can expect to pay for when getting set up. If a company doesn’t respond to our pricing scenario with a specific quote, we estimate the cost of their system based on information taken directly from their websites.
The algorithm for the Most Affordable report is outlined below:
- Feature Set [40%]
- Total Cost of Ownership [40%]
- Customer Reviews [20%]
Our research for our affordability reports is largely based on answers from vendors and information we pull directly from their own sites. However, for our Most User-Friendly reports, we perform direct user testing, diving into the systems to evaluate efficiency, effectiveness, and intuitiveness so we can see for ourselves how solutions perform. To do this, we come up with a list of basic tasks to perform in a system and create a testing handbook, complete with templates to plug into the system so that every testers’ performance is uniform and fair.
We time ourselves to see how long it takes us to complete each individual task in each system and count how many clicks it takes until a task is completed. After we’ve completed all the tasks, each tester gives each system an SUS score, a ten question industry-standard questionnaire that measures user satisfaction.
We complete two rounds of testing in each system. First, we evaluate a system “blindly.” We jump right in without any training to measure how intuitive a system is. Are we able to figure out how to make things work right away, or do we have to click around and explore before completing our task? For the second round, we evaluate a product’s training resources for a given task. After reading an article from an FAQ section or watching a brief video tutorial, we perform the same tasks in the system a second time, measuring time and clicks. If no training resources can be found for a task, we simply perform the task a second time.
For our Most User-Friendly report, we also measure what services vendors provide for their customers in terms of the following categories:
- Implementation – Setting up and configuring/personalizing a system for a specific company’s needs
- Training – Answering basic questions about how to get started and performing basic tasks
- Customer Support – How to reach a company if a buyer has a technical difficulty (or encounters simple user errors)
The algorithm for the Most User-Friendly report is outlined below:
- Usability [50%]
- Customer Service [30%]
- Customer Reviews [20%]
While our Software Lab testers and researchers do very thorough work in trying to recreate an end-user’s experience, at the end of the day, they are not actual customers of these systems. Evaluating user reviews from across Gartner Digital Markets (Capterra, GetApp, Software Advice) allows us to include the voice of actual customers and consumers who do not find themselves in testing situations but in the trenches of day-to-day business operations, relying on the systems we evaluate to get their work done efficiently. We consider the total number of reviews as well as the star rating of those reviews.
In both reports, reviews count for 20% of the final score, so it’s important to gather reviews to increase your ranking. This will also help your conversion rate for your listings in our general directories, as well.
We hope this post has helped to explain our process in ranking software solutions by affordability and usability. Since this is a new series of reports, we welcome and encourage feedback, questions, concerns, and comments. Feel free to email us to help us make these reports thoughtful, useful tools for all software buyers.
Although we try our best to answer the questions “Which software is most user-friendly?” and “Which software is the most affordable?” for the majority of users, your system’s capabilities will not always directly match the tasks and features we focus on. We are always working to expand the scope of the categories covered by our reports. Whether you make the list or not, we always inform software buyers that our research is not all-inclusive and encourage them to research and perform their own testing through demos and trials to determine which software is right for them.