Questing for the Magic Metric

by Evan I. Schwartz

Questing for the Magic Metric

by Evan I. Schwartz
Courtesy of Ben Heath

How Superhuman discovered a galvanizing OKR.


Evan I. Schwartz
Published on 10.01.2019

Imagine if you could find one critical success factor—a single metric that could be measured and tracked—that could galvanize your organization to achieve one clear objective. While large organizations may not be able to settle on just one goal post, startups often live and die by having a single-minded focus.

In 2016, Superhuman was a startup of 14 people struggling to launch its first product, an application for email power users that superimposes a faster, more productive interface for Gmail. Founder and CEO Rahul Vohra was questing for the elusive “product/market fit,” a general state where you know that your efforts have created magic in the marketplace, something that venture capital leader Marc Andreessen famously defined as such:

"You can always feel product/market fit when it is happening. The customers are buying the product just as fast as you can make it—or usage is growing just as fast as you can add more servers. Money from customers is piling up in your company checking account. You're hiring sales and customer support staff as fast as you can. Reporters are calling because they've heard about your hot new thing and they want to talk to you about it. You start getting entrepreneur of the year awards from Harvard Business School. Investment bankers are staking out your house."

Those, however, are outcomes, not objectives, and none of them were happening for the San Francisco-based startup. “In the beginning, I wouldn’t say that we were well run,” Vohra says. “We struggled to have a cadence for what we were building.” Worse, the company’s value statements—such as “create delight” and “achieve remarkable quality”—were failing to create focus for employees.

By themselves, those values were too vague to be of much use. “We did weekly or bi-weekly sprints and those worked fine for short-term planning,” Vohra says. “But when it came to goals—such as this quarter we’ll do this, or in six months, we’ll do that—we were unable to reliably set and meet those goals.”

Courtesy of Ben HeathCourtesy of Ben Heath

Looking for a leading indicator

For a solution to this lack of discipline, Vohra turned to the system of Objectives and Key Results (OKRs). “I stumbled upon information on OKRs and for me, it was a sufficiently compelling and articulate description of a process that I thought we could follow.”

It took the team three or four quarters to really get the hang of using OKRs, to the point where they accelerated development. Yet by the summer of 2017, the company still hadn’t launched its product after three years of coding.

Things only started to turn around when Vohra discovered what he calls “a clarifying metric.” Turns out, product/market fit was a lagging indicator. What Superhuman needed was a leading indicator. Vohra was inspired by Sean Ellis, an entrepreneur best known for his early work with DropBox, Eventbrite and LogMeIn.

What Ellis did was turn to users, asking them, simply: “How would you feel if you could no longer use the product?” The key was to measure and track how many of them answered “very disappointed.”

Ellis surveyed almost 100 startups and found a magic threshold: 40%. Companies that failed to achieve product/market fit came in below that number, while those who did came in higher. For instance, Slack reached 51% and became a giant growth story.

Vohra unleashed this survey question on the beta users of Superhuman, and the number that came back was 22%. This wasn’t terrible. After all, more than 1 in 5 users surveyed were saying that they loved and needed the product. “Our team now had a single number to rally around instead of abstract ideas,” he says.

Yet it wasn’t enough to get over that threshold—40% reporting that it was indispensable, the OKR that would make the company sink or swim. Vohra considered this metric to be more on target than a net promoter score (NPS), as it was personal, not just asking whether you’d recommend the product to others.

Courtesy of Ben HeathCourtesy of Ben Heath
Courtesy of Ben HeathCourtesy of Ben Heath

Targeting high expectations

Vohra went deeper into the data. By segmenting the responses—putting more casual email users in one bucket and creating separate categories for job types—Vohra isolated a group he called HXC, for the “high expectations customer.” These are the folks using email for sales, business development, and other mission-critical functions.

Of those, 32% said they’d be “very disappointed” if they could no longer use Superhuman. That still didn’t meet the threshold for product/market fit, but it was getting tantalizingly close. When asked what the key benefit was, these super-users reported “speed.” Many super-users had the goal of getting through hundreds of emails each day and achieving “inbox zero.”

The survey also showed that the number one complaint about the product was “lack of a mobile app.” The startup was assuming that email power users were desktop user, but that assumption turned out to be faulty. 

By introducing a Superhuman mobile app and optimizing the product around speed, the golden OKR jumped to 58%. By this point, users had been joining long waiting lists to use the product, as raves became the norm. “No app has changed my business life like the week I’ve now spent with @Superhuman,” superuser JP Schust tweeted recently. “It’s impossible to overstate the value this has brought to my ability to deal with email and focus.

In short, many of the product/market fit results highlighted by Marc Andreesen are coming true, leading to another $33 million in venture capital investment at a far higher valuation.

Powering through a privacy crisis

This past July, however, Superhuman was confronted with an immediate crisis. Trade journals were reporting that those who receive email from a Superhuman user are being tracked using on-screen pixels as to what they are doing online—known as read statuses—and even where they were located when they opened their email.

The uproar threatened to derail Superhuman’s progress, as location tracking in particular has become controversial throughout the industry as a potential privacy violation.

To his credit, Vohra responded immediately both in the press and on the company blog, apologizing and promising to change the company’s privacy practices but without doing away entirely with pixel tracking and read statuses, which have become essential for super-users.

Effective immediately, he wrote, Superhuman will stop tracking location, will delete existing location information, and will turn off read receipts by default. “I have come to understand that there are indeed nightmare scenarios involving location tracking,” Vohra said, adding later: “I wholeheartedly apologize for not thinking through this more fully.”

Vohra won praise for his quick and decisive response, making pixel tracking an opt-in feature while not caving to the pressure in the press and retaining read status information.

Courtesy of Ben HeathCourtesy of Ben Heath

OKRs for the future

By now, the company has expanded its use of OKRs beyond the 40% “very disappointed” key result for super-users who are asked what would happen if they lost access. “We’re still tracking it,” Vohra says, “but the plan to make it into more of a health metric going forward.”

The firm is now focused on using OKRs to help scale the company and to turn core values into objectives, such as:

Objective: To refine the entire Superhuman experience to Remarkable Quality

Key Results: 1. Grow to $X annual recurring revenue by the end of the quarter from current: $Y. 2. Desktop: improve sentiment to 92% from current 89%. 3. Mobile: improve sentiment to 75% from current 69%.

Now with 35 employees, Superhuman’s use of OKRs has been flexible enough to help drive new phases of growth, while also aiming for individuals to achieve more personal objectives.

“As we bring on more seasoned managers,” Vohra says, “we’ll also bring on the discipline required to run an individual OKR process.”

Yet it was finding the magic metric for user passion turned out to be the key to its current success. “The use of OKRs has been a game-changer for our company,” Vohra says. The waiting list for Superhuman is now around 180,000 people and social media is abuzz with potential users seeking referrals to jump the queue. That’s when you know you’ve achieved the elusive product/market fit.

Evan I. Schwartz (@eischwartz) is an author, journalist and storyteller who writes about innovation and leadership. Special thanks to Elizabeth Dunne for reporting on this story.

Courtesy of Ben HeathCourtesy of Ben Heath

MORE

article

How 23andMe reinstituted OKRs to power its current phase of growth.

video

Ryan Panchadsaram, co-creator of WhatMatters.com with John Doerr, explains the basics of the OKR (Objectives and Key Results) goal-setting method.

article

A freelancer turns ideas into traction with OKRs.

video

In this video, Alexis Gonzales-Black explains how to write good OKRs with examples so you can start writing ambitious goals, too.

article

How Allbirds sets ambitious metrics for R&D.

article

How OKRs are creating MASSive change.

article

How a company without managers gets where they need to go.

article

An office-wide excel spreadsheet transparently levels the playing field at Apartment Therapy.

article

Minerva Schools is redefining what a university can be by using OKRs to illuminate the future of higher education.

article

Allbirds cofounder Joey Zwillinger talks about measuring sustainability and their carbon footprint by using OKRs.

article

Learn How Possible and Nyaya Health are growing a global culture of collaboration with OKRs.

article

Possible has improved healthcare in Nepal by using BHAGs to maximize their OKRs.

article

Learn how WINS meetings give everyone at Apartment Therapy a sense of collective accomplishment within an OKR cycle.

article

Follow along as the digital publication Apartment Therapy uses OKRs to get to the next level.

article

Thaddeus Ferber of the Forum for Youth Investment nonprofit shares how OKRs encouraged bold moves during a time of budget cuts.

article

Laszlo Bock explains the evidence behind why bonuses should be kept separate from individual goals.

article

Code for America's Jennifer Pahlka and Lou Moore explain how the nonprofit uses OKRs to defeat local government bureaucracy.

article

How transparency makes an organization’s objectives clear and how it can be used in nonprofits, schools, and government.

article

Using OKRs to power growth, engagement, and diversity

article

And Why You Should Too.

article

Using OKRs Even When Your Boss Doesn’t.

video

John shares his take on what makes the difference between success and failure. Telling stories of ambitious leaders and teams, John's keen observations and insights bring light to an oft overlooked aspect of dreaming big.

video

The founder and CEO of 23andMe shares the ups, downs, and in betweens of goal setting.

article

A Silicon Valley luminary finds a new approach to defining success.

article

A closer look at the man who invented OKRs.

article

Meet the California grade schoolers charting their own path.

article

Founders share the benefits and challenges.

Join the community

Join the community

Get exclusive guidance from John, Ryan, and the What Matters team by signing up for our weekly newsletter, Audacious. You’ll learn week-by-week how to sharpen your OKRs and stay on track to reach your goals.

Get exclusive guidance from John, Ryan, and the What Matters team by signing up for our weekly newsletter, Audacious. You’ll learn week-by-week how to sharpen your OKRs and stay on track to reach your goals.