Measuring is good. Metrics are essential for knowing deeply your product and for making more accurate decisions. But you must be very careful. The excess of metrics can get in your way and overshadow your full knowledge of your product.

Data-driven

Instead of excessively collecting metrics, you should follow a data-driven approach. When you manage a product in a data-driven way, you run experiments that generate data that feed your decisions on the next experiments. It is this simple, and it seems to be an excellent way to manage a product. But there a few problems within this approach.

1) Easy versus most relevant

There’s a natural tendency to measure what is easy to measure, and that can happen to the detriment of measuring what is most relevant. For instance, what is easier to measure: clicks in a message from an e-mail marketing campaign or the user perception (curiosity, happiness, disdain, etc.) when getting this message from the e-mail marketing campaign? 

Before taking actions over the existent data, it is always good to ask if these are the best data for making the decision.

2) Local optimization

Another concern in relation to decisions based on data is the hazard of sticking to the improvements focused on local optimization. That is, you make incremental improvements in your product focusing on improving a given metric, but you donít realize that if you make a more radical change you can get a considerable increase in this metric, larger the largest increase obtained from these incremental improvements. 

3) Quality of data

It is necessary to understand the quality of data, that it, how they are obtained and processed before you get them into the analysis. The results are statistically relevant?

You must have heard of A/B tests, right? It is a test in which you build two versions of something. Next, you set apart the traffic in these two versions and start to measure the behavior on the two versions according to a given goal, such as clicks on “Buy” button. After running this test for a while, you will know which of the two versions reaches the goal more often.

Now try to run an A/A test, that is, an A/B test in which the two versions of the page are identical. After running the test, check out the results. Chances are you will find one of the A versions of your page reaching the goal more times than the other one. How is that possible? Statistical variations, seasonality, and unpredictable effects. 

Every data can be subject to statistical variations and that is why. In quantitative data, it is always good to understand the size of the sample and how it was collected. The season when the data sample was taken also affects the data quality: time of the day, day of the week, day of the month, the period of the year, and so on. Data are a photography with a date and time set, and the same datum collected minutes later can give you different information. 

4) Qualitative knowledge

Not only from quantitative data lives the product management. Au contraire, the product manager must use different tools to learn more about the user, the problem afflicting this user, the context in which this happens and what motivates having this problem solved. Many of these tools present qualitative data, that is, exploratory research data, through which it is seeking to understand reasons, motivations, and opinions. This kind of understanding is very difficult, if not impossible, to obtain from quantitative data analysis. 

Data-informed

Instead of data-driven, we need to be data-informed, in other words, using data as one more input to decision-making, not the only input. Take the experience, the intuition, the judgment, and the qualitative information into consideration, along with the metrics to increase the quality of decision-making. 

One of the best examples I can quote has to do with the website hosting product from Locaweb. Through the years, in a reasonably informed way, but always counting on a lot of intuition, we altered our hosting plans in order to have more disk space, data transfer and the number of sites that could be hosted in each plan. In 2011 we noticed that more than 90% of our clients were choosing the basic plan because it attended the needs of most people who needed a website.  

We wanted the largest plans would play a bigger role in sales, but with the limits we had, there was no motivation for clients to buy them. We thought of changing the plans for new subscriptions, decreasing the limits to incentivize clients to get bigger plans. However, as this was a significant and very sensible change, we brought in a consulting expert in pricing that helped us collecting and analyzing several data, to suggest which was the best plan structure change to be done.  

We implemented the changes suggested in 2012. There was a little variation on the distribution of plans, but the number of plans acquired per month didn’t change. Moreover, it even decreased a little, which resulted in none alteration in the amount of monthly revenue. That is, we spent time and money collecting and analyzing data that made us take a decision that didn’t change the company’s result. Maybe if we had defined the changes more intuitively we would have saved money and would know the result of the decision more quickly.  

A little bit on A/B tests

There are two great tools for A/B tests – Visual Website Optimizer and Optimizely. I used the services from Visual Website Optmizer, that gave you one free month to test some hypotheses about the home of ContaCal.

For those who don’t know, ContaCal is a digital product I created in 2011 during nights and weekends with no connection with my day job in order to experiment with building a startup using the most recent software development and product management methodologies and best practices. ContaCal is a virtual food log that counts calories with a twist. Besides telling you the number of calories you’ve ingested it also tells you the quality of these calories.

In less than 30 minutes, I was able to create 4 versions and began to run the test. I decided to test two things. One was the color of the “create account” button to see if it would make a difference in the number of people who would click on it:

The second test: if changing the explanatory video for a photo would increase or decrease the number of people who would click on the “create account” button:

The result was that the photo of the healthy family had more clicks in the “create account” and the one with the woman measuring her waistline had the worse click-through rate:

After seeing this result, I got the impression that if I put the green button with the healthy family picture I was going to increase the conversion even more. So, I decided to run this test and the result was that, with the healthy family photo, the green “create account” button was the worst, and the best option was the blue button:

Therefore, take care, appearances can be deceiving! Make experiments before taking conclusions!

Analysis paralysis

Lastly, aside these precautions, it is necessary to take care with the analysis paralysis effect, that is, analyzing data all the time and not taking any action. As seen on the picture from xkcd.com, analysis paralysis can cost you a lot:

Final thoughts

The metrics are not the reason for the existence of your product. The users, their problems and the strategic goals of the business are the reasons for its existence. In other words, metrics are a mean and not the end, the results, and the goal. Therefore, use them more like one of the tools to help you to drive your product in the right direction.

With this, we close the theme on the growth stage of a software product life cycle. We understood how to deal with client feedbacks, and what it is and how to prioritize a roadmap. We also saw several types of metrics, including the conversion funnel, engagement, churn, global and individual financial metrics, revenue and negative churn, NPS, the loyalty metric, and we also approached some considerations on metrics. 

In one of my next articles, we will understand better the next stage of a software product life cycle: maturity. 

Digital Product Management Book

Do you work with digital products? Do you want to know more about how to manage a digital product to increase its chances of success? Check out my book Product Management: How to increase the chances of success of your digital product, based on my almost 30 years of experience in creating and managing digital products.