Wenatchee Washington is the “Apple capital of the world.”  Between September and October, you can see trees heavy with ripe apples and many orchard visitors picking u-picking apples. What is not easily observed is all that goes into maintaining a healthy tree and the most delicious fruit. The right amount of sun and moisture, as well as the level of acidity in the soil. Even though fruit hangs heavy on its branches, the underlying health of the tree, not yet visible, may be compromised. Data can suffer from similar observability bias. The most visible and easily collected data does not guarantee accuracy or results. Metrics that appear obvious and are easily collected, while convenient, may not provide the insights required. Observability bias would prove to be an issue for Dropbox search. Search success was measured based on the customer clickthrough rate of search results. Clicks into any of the search results was counted as a successful search. It seemed logical and provided an easily observed customer behavior, yet Dropbox search launched feature after feature without significantly improving the metric of success. The leadership team wanted to validate, and improve, the way search success was measured.

The importance of having the right metrics cannot be understated. The search success metric informed the prioritization of work across multiple teams and provided the foundation for the strategic roadmap. In order to give the team a balance of tactical and strategic insights, my initial project for search combined observational interviews and an in-line survey, triangulated against customer analytics. The research was completed in multiple stages, providing actionable recommendations after each phase. Customer interviews, including direct observation of the search experience, highlighted critical customer pain points. An inline survey collected self-reported customer success. The qualitative interviews provided insight on which customer behaviors were the most important to triangulate against success. The data triangulation of self-reported success and analytics exposed inflated search success rates and informed the development of a new metric of success. Finally, a cross-functional team workshop surfaced strategically important customer job stories.