Beloved metric and NPS analysis
Context:
At Newsela, we used multiple metrics to measure product satisfaction and engagement. However, some of these metrics had challenges, were duplicative, and/or were incomplete.
A discovery sprint, followed by user interviews and statistical analysis, was conducted to validate using NPS as a beloved metric.
The graphs and data presented are either part of the original UXR reports or created to present the project and key information is blocked to protect the confidentiality of the research. Objectives:
Design a key performance indicator that measures user (teachers and admins) satisfaction and brand love and is easy to implement, analyze, understand, and track over time. Avoid conducting, analyzing, and sharing multiple metrics that could be time-consuming and repetitive. Our goal was to design a survey instrument that captures the beloved metric, a design that significantly influenced the final outcome.
My Role:
Key Learnings:
Impact:
My responsibilities included:
Research Design: Designing a brainstorming workshop with a cross-functional team, including product marketers, consumer success managers, educational researchers, and a leader from the product team to design the survey instrument that the captures beloved metric.
Data Collection: Lead team to distribute surveys and conduct interviews. interviews, and to gather detailed insights into user experiences and perceptions.
Analysis: Partnering with statisticians to analyze and compare our proposed metric and NPS results.
Reporting: Presenting regular updates to the findings to stakeholders through detailed reports and visual presentations.
Collaboration: Working closely with cross-functional teams, including their experience during the workshop and analysis of the data.
Performance: Psychometric analyses showed that the questions all perform well in measuring aspects of Belovedness.
Beloved metrics correlation with NPS: NPS positively correlated with all of the items successfully but not too strongly: this is good news as it validates that the Beloved items measure something similar but not identical to NPS, which is desirable.
Composite Metric: Created a composite metric by taking the mean of the four items. With a mean and median of 4 (out of 5), the data are skewed toward higher scores.
Cross-functional collaboration: The project fostered a culture of collaboration across departments, ensuring that everyone from product managers to customer support teams was aligned on the importance of user satisfaction and loyalty metrics.
Validation of NPS: We identified that NPS closely reflected the beloved metrics. This finding confirmed the accurate use of NPS as a good metric for product satisfaction. By continuing to use NPS, we avoided losing valuable historical data.
*For additional findings and learning, please contact svarchmalena@gmail.com