Intermediary Metrics: Goaling, Not Gaming

All metrics are susceptible to gaming. If any team’s mission boils down to a handful of metrics, they may optimize those metrics at the expense of the company’s larger goals, often unintentionally. Aligning incentives and metrics with the company’s core mission is perhaps the most important job for a growth leader to take on.

Mike’s first job at Stitch Fix was to build the user acquisition function from the ground up. There was a separate team that focused on retention. And as he did so, he realized that there was tension between acquisition and retention teams. It came down to the metrics they were each trying to optimize for.

“Because we’re a subscription-based service, you want to optimize for clients who are going to stick around, and not be shortsighted on first-time transactions… [yet] an acquisition team would send a bunch of shitty signups through”, which makes sense because their target metrics were around solely acquisition. Gaming isn’t always intentional, but it’s a natural consequence of teams focusing on a single KPI over the big picture.

In order to set his team up to focus on the types of leads that would, over the long term, be good for Stitch Fix, Mike had to find a way to incorporate retention into his acquisition goals.

Are You Gonna Pay Us Again?

Mike’s team at Stitch Fix found a great data point to inform lifetime value prediction and align acquisition and retention teams. They introduced the question, “Are you anticipating your next fix?”.

This straightforward question became an excellent predictor of customer retention and 12-month value, which the company started using as a key proxy for a user’s lifetime value. In a business model driven by subscriptions and customer loyalty, this figure was essential to understand and optimize.

Mike said the idea came from his “Better Data Team”, three PM analysts whose entire job was to optimize the style profile and checkout flow. They relied heavily on heuristics to provide the best predictive questions over time.

Superhuman asked a similar question to gauge product indispensability: “How upset would you be if the product was taken away?” Questions like these are an excellent gauge for customer loyalty and the value you add. Don’t be afraid to ask your users for this information.  

Always Go Deeper  

The key idea that drives Mike’s philosophy for data-driven growth is this: always go deeper than the metric itself. Seek to understand the incentives and confounding variables that could take you away from what you really want: efficient, sustainable growth and fully aligned efforts.

Mike’s biggest suggestion: to build a framework for your company grows and assign each step a relative importance. He visualized this idea as a “Growth Diagram”, which should cascade down from the north star goal of the company to each individual and the metrics he or she is responsible for. This not only makes individual roles and contributions clear, but tensions between metrics will become apparent as well.

One last strategy that Mike recommends: play defense ! As we’ve discussed with other industry leaders, you need to revisit your model often to refine it, particularly when you hit bumps in the road. Mike described his Model Adjustment Hour, a periodic meeting that incorporated data and qualitative feedback. Collective understanding and response to issues, such as a metric dropping or a system-wide outage, creates a more robust, resilient organization.  

Aligning Goals, Big and Small

If you’re a contractor, and your goal is to build a house, you’re going to need bricks to do the job. If you give your team only the goal of finding as many bricks as they can, you’ll end up with a shaky house haphazardly constructed. You need everyone to understand the bigger picture, and to create a system of smaller goals and incentives that drive the whole team toward the same mission.

Mike’s key wisdom is to look beyond any single metric, to see the relationship between the different aims and actions of your business, and to always align all efforts with your company’s north star.

‍