How to start a Product Outcomes Measurement Program.
Who is this written for?
This article is for product managers, product designers, user researchers, software engineers, data scientists and quality engineering team members. Basically, everyone in a product team can and should be aware of how to measure the performance and impact of the products they create. This article is meant to serve as a guide to introduce product measurement strategies and the theory behind which analytics to choose to measure an outcome.
So, what exactly are Product Outcomes and how can you measure them?
A product outcome is an observable and measurable metric, data point, action, behavior, financial, or business variable that helps product team members understand if the product, service or projects they ship are delivering value to their customers or to the businesses they work for. Product outcomes and their corresponding analytics help you measure in real time or over time the efficacy of strategic and tactical decisions made by your team.
How can measuring Product Outcomes help you ship higher quality products?
Measuring product outcomes allows product team members avoid the “build trap” that most tech companies fall in. Product teams that lack clear measurements of success and an understanding of customer behavior in their products generally continue to build features of dubious value in an attempt to engage customers, often with diminishing returns. Measuring product analytics also enable product team members to diagnose where pain points might exist, and surface gaps in your products value propositions. By measuring product analytics and outcomes help decision makers understand how much value or how useful a product is.
What are the different types of product metrics you can use to measure product outcomes?
Outcome metrics can be measured using a variety of analytics that track different events, behaviors and activities. Standardizing on which methods of product outcomes to track ensures alignment across business and a shared and vision of success.
Behavioral metrics or product management analytics is the process of sorting and counting behaviors in order to construct behavioral cohorts to analyze things like engagement, conversion, and retention, as well as measure how behaviors change over time.
Qualitative metrics or sentiment analytics capture is a natural language processing technique or numeric qualitative response score used to determine whether data is positive, negative or neutral. Sentiment analysis is often performed on textual data to help businesses monitor brand and product sentiment in customer feedback and understand customer needs.
Business metrics use statistical methods and analytics to analyze historical data in order to gain new insight and improve strategic decision-making, Business metrics primarily measure profits gained or loss or overall business growth.
Leading metrics are any measurable or observable variable of interest that predicts a change or movement in another data series, process, trend, or other phenomenon of interest before it occurs.
Lagging metrics track the observable or measurable factor that changes sometime after the economic, financial, or business variable it is correlated with changes. Lagging indicators confirm trends and changes in trends. Lagging indicators typically measure variable changes over an extended period of time, the periodicity is determined by the businesses objectives and desire to respond.
How can different product team roles use product analytics to inform their product efforts?
Product Managers and team leaders can measure business, performance, and customer outcomes. Product managers often construct custom dashboards to measure performance for their business area, feature set, or value chain to stay on top of their products performance.
User researchers can measure the provenance and location of user frustration or with equal measure delight moments within user journeys, through step funnel analysis and or sentiment analysis analytics.
Software and engineers can measure and monitor system performance, resource availability, and latency across sub systems. Environment analytics are often a great source of indicator metrics that can tell you overall how well the system is performing and can help isolate root causes of problems, that can affect the customer experience.
Product designers can measure the behavioral, task completion rate, and user sentiments analytics to gauge the efficacy of new features and redesigns.
What are the types of measurable product outcomes or analytics I can choose from?
Operational Metrics (Leading Indicators)
Usage over time
The amount of user activity in your product over time, to see if the observed activities are decreasing or increasing overt time.
How do you measure it?
Repeat logins/ session instances over time, you can measure usage weekly, monthly, and daily active users.
Stickiness
You can track stickiness by finding user originated activities and track the usage of a product or feature.
How do you measure it?
You can track the number of Daily Active Users (DAU): DAU is calculated as the total number of unique users who use your product on any given day. In other words, DAU is the total count of users who use your product at least once in a day. Monthly Active Users (MAU): MAU is calculated as the total number of unique users who use your product in a month. In other words, MAU is the total count of users who use your product at least once in a month. An active user can be anyone who logs into the product, and completes a task
Feature Adoption Rate
The percentage of new users of a feature.
How do you measure it?
The formula for calculating adoption rate is: Adoption rate = number of new users / total number of users. For example, if you have a total of 1,000 users, of which 250 are new, then your adoption rate is 25% (250/1,000).
Feature Retention
Feature retention tells you how valuable a feature is to your users. answers the question “do people come back to use this again and again?” That is, do people who viewed this feature last week continue to come back and use it today?
How do you measure it?
Create a cohort for the purposes of user retention, cohort analysis. Overtime usage for new features trails off after an initial burst of user adoption.
Product Availability
Product availability metrics measure the emergence, duration, and impact of outages that occur in a product overtime
How do you measure it?
Measure the frequency and duration of outages or inoperable systems over time.
Product Defects
Defect rate is calculated by testing output for non-compliances to a quality target. … Product engineers may calculate multiple defect rates based on different … Software testing tests 500,000 lines of code and discovers 33 defects.
How do you measure it?
This can be calculated by measuring user sessions that experience defects and tracking the total, frequency and location over time.
Task Completion
Task completion is one of the most common ways to quantify the effectiveness of an interface. If users can’t do what they intend to accomplish, not much else matters.
How do you measure it?
Tracking task completion starts by tracking the start of a task divided by competition of a task by a group of users over time
Qualitative Metrics/ Sentiment Analysis (Lagging Indicators)
Product Market Fit analysis
Product-market fit describes a scenario in which a company’s target customers are buying, using, and telling others about the company’s product in numbers large enough to sustain that product’s growth and profitability.
How do you measure it?
You can ask your uses: “How do you feel if you could no longer use the product?” Evaluate your answers on three-point scale: very disappointed, somewhat disappointed, and not disappointed at all.
Likert Scale
A Likert scale assumes that the strength/intensity of an attitude is linear, i.e. on a continuum from strongly agree. Likert scales are used to structure customer sentiment and attitudinal feedback in a survey.
How do you measure it?
Create psychometric survey questions that are response scales in which responders specify their level of agreement to a statement typically in five points: (1) Strongly disagree; (2) Disagree; (3) Neither agree nor disagree; (4) Agree; (5) Strongly agree.
Customer Satisfaction Score
Customer Satisfaction Score (CSAT) is the most straightforward of the customer satisfaction survey methodologies, and it measures customer satisfaction with a business, purchase, or interaction.
How do you measure it?
It’s calculated by asking a question, such as “How satisfied were you with your experience?” There’s a corresponding survey scale, which can be 1–3, 1–5, or 1–10.
Customer Effort Score
Customer effort score is recorded on a numeric scale, so a higher score would represent a better user experience
How do you measure it?
Customer effort score is recorded on a numeric scale, so a higher score would represent a better user experience. For a standard seven-point scale, responses of five or higher would be considered good scores. For a standard seven-point scale, responses of five or higher would be considered good scores.
Strategic Business Metrics (Lagging Indicators)
Revenue calculated as MRR or ARR
Revenue is the key driver of growth; this can be calculated monthly or annually.
How do you measure it?
Measure the amount of revenue generated over period of time or interval.
Conversion Rate
Measure of potential nonpaying customers to paid customers.
How do you measure it?
Measure the number of non-paying customers that completed some activity to become a paying customer. Conversion is usually measured as a percentage.
Customer Acquisition Cost
CAC is the total cost to acquire a customer. This is calculated by totaling the total cost of sales and marketing efforts. Product driven businesses strive to drive down the cost of customer acquisition overtime to reduce cost and spend.
How do you measure it?
You can determine your customer acquisition cost by adding all costs associated with your customer acquisition efforts (marketing costs and sales efforts and dividing that by the number of new customers you acquired as a result.
Lifetime Value (LTV)
Lifetime value is the measure of the future potential revenue associated with a customer which is modeled based on revenue and expansion assumptions.
How do you measure it?
Calculate the “landed” revenue than multiply by the expansion assumption based on cross sell and up-sell opportunities.
Net Revenue Retention (NRR)
This is the measure of percentage of change in recurring revenue from your pool of customers. NRR should be north of 100%. This churn metric gives a comprehensive view of positive as well as negative changes with respect to customer retention.
How do you measure it?
Calculate the percentage change in recurring revenue from your pool of customers.
Gross Margin
Gross margin is a company’s net sales revenue minus its cost of goods sold (COGS).
How do you measure it?
To calculate gross margin, subtract Cost of Goods Sold (COGS) from total revenue and dividing that number by total revenue (Gross Margin = (Total Revenue — Cost of Goods Sold)/Total Revenue). The formula to calculate gross margin as a percentage is Gross Margin = (Total Revenue — Cost of Goods Sold)/Total Revenue x 100.
Profitability
Profit margin or percentage lets you know how much profit your business has generated for each dollar of sale.
How do you measure it?
Net Profit Margin = (Net Income / Revenue) X 100
Net Profit Margin = [(Revenue — COGS — Operating Expenses — Other Expenses — Interest — Taxes) / Revenue] X 100
Gross Margin = [(Total Revenue — COGS) / Total Revenue] X 100
Win Rate
Win-rate is calculated by dividing the number of sales opportunities converted into successful deals by total number of opportunities.
How do you measure it?
Calculate your winning percentage simply get the total number of sales by adding your wins and losses together. From there divide your number of wins by the total number of losses.
How do you build a good product measurement strategy?
You need a combination of leading as well as lagging indicators to understand how your product is performing in real time as well as help you identify the direction of top-level trends. The indicators you select should tell you how well you are doing, where are you doing well, and how much value you are delivering to customers and your business. Well-constructed product outcome strategies have a mix of both business and product/ behavioral metrics. There is no ideal number of metrics one should measure at one time but a just enough strategy can help you and your team funnel metrics
What are some of the leading platforms used to measure product outcomes?
Most product and business platforms have similar functionality but differ in installation, methodology of tracking, pricing models, and 3rd party integrations. Considerations for product team organizations when selecting an analytics platform should be the ease of implementation, number of end user seats. Whatever solution(s) you choose want a large number of seats that the whole team/ department can gain visibility.
-Amplitude
-Google Analytics
-Pendo
-Qualtrics
-Heap
-Salesforce
-Mixpanel
-Klipfolio
References
Davenport, T. H., & Harris, J. G. (2007). Competing on analytics: The new science of winning. Boston, Mass: Harvard Business School Press.
Olson, T. (2020). The Product-Led Organization: Drive Growth By Putting Product at the Center of Your Customer Experience (1st ed.). Wiley.
Cagan, M. (2017). Inspired: How to Create Tech Products Customers Love (Silicon Valley Product Group) (2nd ed.). Wiley.
Bland, D. J., & Osterwalder, A. (2019). Testing Business Ideas: A Field Guide for Rapid Experimentation (The Strategyzer Series) (1st ed.). Wiley.
Empower Product Teams with Product Outcomes, Not Business Outcomes. (2020). Product Talk. https://www.producttalk.org/2020/05/product-outcomes/#:~:text=To%20clarify%20the%20difference%20between,is%20moving%20the%20business%20forward
