Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
Developer sites such as Stack Overflow have exploded in popularity with over 125 million active users per month; however, the large majority of questions are never answered. Vendor community forums lag further behind with significantly lower activity levels leading to costly support cases with average time to resolve of 14 days. As the pace and complexity of technology increases, our view is technical support needs to fully embrace community forums as the best place to get answers.
The reality is technical people do not want to talk to a support bot and view opening a support ticket as a last resort. They want answers quickly from someone they trust and their questions are generally rich in technical detail. While today’s forums can be great gathering places to pose technical questions and connect with experts, answers provided by the community are on a “best effort” basis and quickly become out-of-date.
Our company analyzed more than 50 forums, including 12 million unique questions and responses spanning enterprise vendors, open core software and cloud-native open source distributions. Our goal is to understand how “best effort” communities perform in solving technical questions and identify the best-in-class forums that set the bar. The results are published for anyone to browse and are regularly updated.
Best-Effort Metrics
The graphic below shows the key metrics for question resolution and how much of the community is engaged in generating answers. This is the average of all the forums in our current data set at the time of publishing.
One key takeaway is that less than one-third of the questions are ever answered and less than 20% resolved in 24 hours. This is a key risk indicator of potential unhappy users and new support cases. A second more surprising result is that only 20% of the community are generating 80% of the answers! This is a clear signal that forums on average are not doing a great job of educating and incentivizing its community members to quickly respond. The most common incentives today are individual reputation ratings and leaderboards for community contributors. Stack Overflow enables users to offer monetary bounties for challenging questions.
The Best-In-Class Metrics
In contrast, the graphic below shows the best-in-class result for each metric.
We were very excited to find a forum with 72% of the community generating 80% of the answers. This feels like a good high-water mark that all forums should strive to attain. On the other hand, the metric of 42% of questions resolved in less than 24 hours indicates a lot of upside potential to make forums the best way to get answers.
So, who is best-in-class? As of this publish date, three of the top forums are open source communities.
- Resolved Forum Posts: KUBERNETES
- Resolved in less than 24 Hours: POSTGRESQL
- Resolved with 1st Response: ALTERYX
- Number of Experts that Resolve 80% of Questions: NGINX
Open source technologies live or die based on high-volume adoption of free distribution, relying on peer support and education to build virality. So it’s not surprising that they rank highly on our benchmark. We did find that the discussions of hottest cloud-native technologies, including Kubernetes and its many variants and tools, happen in many different forums. This includes the hosted forum by the distribution owner (e.g., Google), Stack Overflow (sometimes many different tags) and Slack channels hosted by the Cloud Native Computing Foundation. Conversations beginning on one forum quickly are diverted to another that specializes in that topic.
Who are the Community Forum Experts?
Within each of these community forums, we also classified the members by level of activity in answering questions with accepted results. Our goal is to identify the most motivated experts who have the highest impact on the community experience. Typically, these experts are motivated by the implicit and explicit rewards of their personal reputation on the forum.
The classification metrics for assessing the experts are:
- Replies: Number of replies by an individual.
- Resolutions: Number of resolved posts by an individual.
- Acceptance Rate: The percent ratio of resolved posts divided by replies posted by an individual.
Using these metrics, we created the following classifications:
- Contributor: An individual who has resolved at least one forum post.
- Expert: Contributors that are in the 90th percentile by number of replies and resolved posts.
- MVP (Most Valuable Player): Experts who have posted higher than above average number of replies and number of resolved posts.
- Rising Star: Experts who have posted below average number replies but have above average acceptance rate.
- High Potential: The remaining experts in our classification.
The results for the Cisco forum are shown here as an example, with NGINX following that for contrast. Note the size of each bubble represents the Acceptance Rate (%). For NGNIX, many of the experts have a greater than 50% acceptance rate even in the High Potential classification versus Cisco, where even the MVPs and Rising Stars average around 20%.
Cisco:
NGNIX:
The expert classifications for all forums in our benchmark can be viewed here.
Community Forums: To Best-in-Class and Beyond
The community forum benchmark shows the potential to drive significant case deflection costs by transforming from the current best effort state to best-in-class metrics. We believe the key to unlocking this potential is empowering the experts who have demonstrated motivation with tools to help them answer more questions faster and with high acceptance rates. One promising approach is to apply machine learning to all prior conversations and published content to recommend the most relevant information to assist an expert while they are working on a question. Simply, as more experts create more successful answers, every forum has the potential to meet and beat today’s best-in-class metrics. And as forums thrive, they will naturally grow in question volume to become the primary technical support channel.
