Software Buying Tips

Capterra Research Methodologies

Published by in Software Buying Tips

The Capterra Shortlist Methodology

Current Methodology: January 2021 – Present

Capterra Top 20 is now Capterra Shortlist! Same data-driven methodology; new look. 

With our new Capterra Shortlist reports, we are still identifying the top products in our software categories based on a proprietary blend of user ratings and popularity. Where the Top 20 was a flat list, Capterra Shortlist plots the top products on a chart, allowing buyers to easily see which products are strongest on each component. The chart view acknowledges the nuance and individuality of buyer preferences and makes it easier for buyers to discover the best software for their needs.

Just like a software buyer’s shortlist of products to consider, Capterra Shortlists may include more or fewer than 20 products, depending on the number of top-scoring products in the category—doing away with an arbitrary requirement of our Top 20 reports.

We use exclusive data and trusted reviews from verified software users to build our Shortlists, so buyers can confidently build theirs.

The Basics

The Capterra Shortlist methodology scores vendors on two dimensions: User Ratings and Popularity.

  1. The Ratings component is plotted on the X-axis and considers the overall user ratings of products based on reviews written in the last 24 months. Users rate products on a scale of 1 star (lowest) to 5 stars (highest). 
  2. The Popularity component is plotted on the Y-axis and considers average monthly search volume for a standardized set of keywords for each product, as derived by a proprietary search methodology, as well as the position of the vendor’s domain on the search engine results page (SERP) for each keyword. This is a measure of user interest and relevance used as a proxy for popularity. 

Each score (Ratings and Popularity) is weighted and scaled to a value between 1-50. 

Data sources include approved user reviews, public data sources and data from technology vendors. Please refer to the Capterra Community Guidelines for more information.

Our Capterra Shortlist reports represent a snapshot in time, using data from a defined timeframe. Capterra Shortlist reports should therefore be used along with the detailed and current information available on each product’s profile page for the most up-to-date view of the data.

Inclusion Criteria

To be eligible for inclusion in a category, products must meet the following criteria:

  • Product has at least 20 unique product reviews published on Capterra within 24 months of the start of the research process for a given report. Two years of reviews provides a sample that we have determined is large enough and recent enough to be valuable to buyers. The criteria is also set  low enough to ensure emerging vendors can be represented.
  • Product shows evidence of offering required functionality as demonstrated by publicly-available sources, such as the vendor’s website.
  • Product serves North American users, as demonstrated by product reviews submitted from that region.
  • Product is  relevant to software buyers across industries or sectors—in other words, no “niche” solutions that cater exclusively to one specific type of user, as determined by our analysis of user reviews and/or market research.
  • Product achieves a minimum normalized overall user-review rating, after application of the Capterra Shortlist method for normalizing and weighting review recency (see description under “Scoring” section).
  • Product achieves a minimum popularity score.

Scoring

To arrive at a final Rating score per product:

  • Scores are calculated for the Ratings dimension based on overall average user ratings. 
  • Newer ratings are weighted more heavily to place higher value on recent reviews.
  • Average ratings are normalized by taking into account the average product rating, the average rating of products in the category, and the volume of reviews for the product.
  • Scores for each dimension are scaled to a maximum of 50.

Products are represented, included, and/or scored solely based on user ratings and popularity data and independent of any relationship that Capterra has with vendors.

Information for Vendors About The Capterra Shortlist

External Usage Guidelines

Providers must abide by The Capterra Shortlist External Usage Guidelines when referencing Capterra Shortlist content. Except in digital media with character limitations, the following disclaimer MUST appear with any/all Capterra Shortlist reference(s) and graphic use:

The Capterra Shortlist report constitutes the subjective opinions of individual end-user reviews, ratings, and data applied against a documented methodology; they neither represent the views of, nor constitute an endorsement by, Capterra or its affiliates.

Escalation Guidelines

We take the integrity of our research seriously. If you have questions or concerns about The Capterra Shortlist content, the methodology, or other issues, you may contact the Capterra Shortlist team at digitalmarketsresearch@gartner.com. Please provide as much detail as you can, so we can understand the issues, review your concerns and take appropriate action.

Tentative Publication Schedule

Please visit our full publication schedule here. Note: This schedule is intended to be used as a guide only and is subject to change at any time at our discretion.

Capterra’s Top 20 Report Methodology

Methodology: May 2020 – December 2020

The Basics

The Top 20 methodology scores vendors on two dimensions: Reviews and Web Search Interest.

  1. The Reviews component considers the “overall” user rating (1-5 stars) of products based on reviews left in the last 24 months.
  2. The Web Search Interest component considers average monthly search volume for a standardized set of keywords for each product, as derived by a proprietary search methodology, as well as the position of the vendor’s domain on the search engine results page (SERP) for each keyword.

Each score is weighted and scaled to a value between 1-50. The two scores are combined into a total score (out of a total possible 100 points), which dictates the rank order of products in the report.

Data sources include approved user reviews, public data sources and data from technology vendors. Please refer to the Capterra Community Guidelines for more information.

Our Top 20 reports use a snapshot of review data from a defined time frame and are not updated after publication. Top 20 reports should therefore be used along with the detailed and current information available on each product’s profile page for the most up-to-date view of the data.

Inclusion Criteria

To be eligible for inclusion in a category, products must meet the following criteria:

  • Product has at least 20 unique product reviews published on Capterra within 24 months of the start of the research process for a given report. Two years of reviews provides a sample that we have determined is large enough and recent enough to be valuable to buyers. The criteria is also set low enough to ensure emerging vendors can be represented.
  • Product achieves a minimum normalized overall user-review rating, after application of the Top 20 method for normalizing and weighting review recency (see description under “Scoring” section).
  • Product achieves a minimum web search interest score.
  • Product shows evidence of offering required functionality as demonstrated by publicly-available sources, such as the vendor’s website.
  • Product serves North American users, as demonstrated by product reviews submitted from that region.
  • Product is relevant to software buyers across industries or sectors—in other words, no “niche” solutions that cater exclusively to one specific type of user, as determined by our analysis of user reviews and/or market research.

Information for Vendors About Top 20

External Usage Guidelines

Providers must abide by Capterra’s Top 20 External Usage Guidelines when referencing Top 20 content. Except in digital media with character limitations, the following disclaimer MUST appear with any/all Top 20 reference(s) and graphic use:

The Capterra Top 20 report constitutes the subjective opinions of individual end-user reviews, ratings, and data applied against a documented methodology; they neither represent the views of, nor constitute an endorsement by, Capterra or its affiliates.

Escalation Guidelines

We take the integrity of our research seriously. If you have questions or concerns about Top 20 content, the methodology, or other issues, you may contact the Top 20 team at digitalmarketsresearch@gartner.com. Please provide as much detail as you can, so we can understand the issues, review your concerns and take appropriate action.

FAQs

What are the specific keywords used for the Capterra Shortlist Popularity score? Can software and SaaS providers do anything to improve their scores?

Do software providers need to notify you if they’d like their product(s) to be considered?

Do you need any additional information from providers to create the reports?

Does it matter which of the Gartner Digital Markets sites — Capterra, GetApp or Software Advice — reviews are published on?

When will you publish [x] category next?

Are software providers running paid campaigns on Capterra more likely to place higher in reports?

How can providers improve their products’ placement on reports?

Can you update a product’s placement in a recent report if that product recently received a number of new reviews?

What are the specific keywords used for the Capterra Shortlist Popularity score? Can software and SaaS providers do anything to improve their scores?

The Popularity score is a proprietary approach to capturing key characteristics of end-user search behavior in a way that is not affected by a provider’s SEO prowess. We do not share the specific keywords included, nor changes we make over time. To that end, there are no specific actions for providers to take to improve scores.

We can say that our approach currently takes into account average monthly search volume for a standard set of product-focused keywords, as well as data on average Google search engine results page (SERP) position for the provider’s domain.

Do software providers need to notify you if they’d like their product(s) to be considered?

No, we will automatically consider all products with profiles on our site and that meet our eligibility criteria in a given category.

Do you need any additional information from providers to create the reports?

No, at this time we don’t need nor accept any information directly from our software and SaaS provider community. We obtain all the information we need via our internal databases and publicly available data.

Does it matter which of the Gartner Digital Markets sites — Capterra, GetApp or Software Advice — reviews are published on?

No, reviews are shared across all three sites and all are considered equally in our research.

When will you publish [x] category next?

Please refer to our research calendar for a tentative schedule of reports by month and category. Note: This schedule is intended to be used as a guide only and is subject to change at any time at our discretion.

Are software providers running paid campaigns on Capterra more likely to place higher in reports?

No. A software or SaaS provider’s status as a client or non-client is not a factor in any way.

How can providers improve their products’ placement on reports?

While software and SaaS providers cannot directly influence their position in reports, the best way to ensure a product is considered is to collect more reviews.

Can you update a product’s placement in a recent report if that product recently received a number of new reviews?

Currently, our reports are published snapshots in time as denoted by the specific publish date on the report. We do not make changes after they have been published unless there is a significant error in its production. Any such products and new reviews will be considered when we next update the report. Please refer to our research calendar for our tentative schedule of reports by month and category.


About the Author

Capterra

Capterra

At Capterra, we believe that software makes the world a better place. Why? Because software can help every organization become a more efficient, effective version of itself.

Comments

No comments yet. Be the first!

Comment on this article:

Comment Guidelines:
All comments are moderated before publication and must meet our guidelines. Comments must be substantive, professional, and avoid self promotion. Moderators use discretion when approving comments.

For example, comments may not:
• Contain personal information like phone numbers or email addresses
• Be self-promotional or link to other websites
• Contain hateful or disparaging language
• Use fake names or spam content
Your privacy is important to us. Check out our Privacy Policy.