Jonathan Sadow

um help me just get away of the land so how many people are at a company under 50 people all right cool 50 to 150 more than 150 it's actually pretty spread out okay awesome that'll help me understand a little bit about where maybe people are coming into thinking about retention thing about growth metrics a little bit about me I know when I've been in the audience which is most of the time hi my conference experience I always wonder like who is this person and why do i why do I need to take anything from them I'm a dad I have two kids I'm from Atlanta originally and I worked at Google for a while until I left Google where I was a product manager at the end of my time there to start to start scoop which I'll tell you about in a little bit all right cool so well time in so we're gonna talk about retention metrics this track is retention and one of the disclaimers I'll give up front is that well I will talk about new retention metrics I am NOT gonna be so bold as to pretend that I'm about to show you some groundbreaking thought on retention that you have not read on Andrew Chen's blog or that you have not been to another session about really instead when I say it's time to find new retention metrics it's really a challenge about how you and your company think about retention right we went through this challenge at scoop and I'll talk a little bit about that it's really a question of how do you think about what it means to measure retention in your own company versus hey let's talk about an industry changing paradigm around retention somebody today might have that or in this week but that that will not be me so I'll talk about scoop very briefly just you have some context so scoop is a carpooling company we are the largest carpooling company in the in the country we partner in enterprises to connect their employees with other co-workers neighbors nearby companies to share their commute to work and so literally we are helping people not feel like they are beholdin to their solo drive to work so that they can turn the morning into an empower 'full meaningful experience and the same with the way home and so when we think about users and growth and retention we're literally thinking about how do we help people choose to either ride or drive a co-worker a neighbor on the way to work it's it's on a surface level very straightforward very simple the one thing I'll super clarify so that it makes sense when I talking about retention is everybody's just a person meaning everybody's just an employee or a neighbor where there's no paid drivers who don't have a network these are just folks who decide to either use their car or get a ride to work with a co-worker somebody that they might know from down the street and so to just give you a little bit of sense of sort of what growth is looks like it's scooped in the three and a half years since we did our first trip we've done more than four million trips across the country I had something like coming up on 70 million miles that people have shared on the way to work these are longer commutes as you might imagine the average commute in the United States is approaching 20 miles so you talk about people who are driving summer of a summer over several new city and they spend a lot of time in the car together which means there's a lot of nuance and what it means to understand how they enjoy an experience and what retention looks like and we work with some of the foremost companies in the country I won't read their names off you could read them but we've had to think long and hard about what does it mean to be a valuable part of the experience when you work at a LinkedIn a best place to work type company and how do we fit into that to build an experience and a product that will retain users so I'll talk about retention for a minute and growth this is not groundbreaking I hope and I know everybody in the room will come into this from different points of view of how you think about growth in general I think that especially over the last couple of years we've clarified that our point of view on what growth means even though there is a growth track and a retention track here which I'm still trying to figure out but I'm happy to still posit that I think when we think about growth we think about how do we bring users in how do we get them to do the action or the behavior or the thing in our app or experienced website etc that we want them to do and how do we get those users to keep doing that over time pretty straightforward and over the last few years as you can tell that attention is its own track we've really started to focus on the retention side of growth versus maybe just throwing dollars in ad campaigns or flooding the top of the funnel right I'm sure that's something that every one of you have talked to your team's about at some point in terms of how do we think about the long term value of the users that we are retaining and so the thing about user growth is that we have lots of different ways to think about how we measure what user growth many of you are looking at ma use W a used daily active users whatever it is that tells you how many people are doing this thing that you want them to do every week that's fine that's good for a top-line metric it's probably on a you know a lot of TV somewhere in your office or it's in some report that your CEO or your your growth folks have to read you guys probably think about it a lot but it only really tells us how many people are doing a thing and it's a little bit removed from maybe where we're spending our energy and I'll come back to that in a minute this the next the next group here is around cohorts and what I would call a cumulative charts or how much about how much something is happening over time right to me and maybe to many of you cohorts are sort of at the cornerstone they're at the they're kind of the holy grail of how we should think about our jobs as marketers product people engineer as growth growth minded people they help us understand overtime are we doing something better in this wave or in this feature or in this process than we were before and are we keeping those people better over time right so flash a cohort turn up it's the best way to possibly know what you might be working on how it affects the user the challenge though is all of this stuff is really user driven it's all about the unit of a person doing do just being in existence right did they activate are they retained how can I tell that more people from this cohort are gonna be using in month three than the last cohort who were using in month three but our job as growth-oriented myath or ants at growth minded folks whether it's marketing or product is to create value right now I know that many of you have different points of view on what the value is so some of you if you're an e-commerce anybody who works in e-commerce right value is money right like generally speaking like you want people to buy things you want to make money if you're in something social or gaming you also want to make money eventually but there's many many things on the value chain that you probably think about and talk about as proxies to get to that value but ultimately we're building companies to either and or create impact to make money right that's that's ultimately what we're trying to do is affect the world that we're invested in and put dollars back into the company or into the share holder so we can keep doing that thing but when we look at users often we're blind about the value that we're creating and we just say are more people doing this thing and we forget what that actually means in terms of value I don't wanna pick on them but you think about Twitter right like easy example of you can have tons of users and actually over time think about what it means to create value that usually gets wrapped up in something called monetization which I'm sure many of you are thinking about going to those sessions etc but monetization ends up being this sort of solution to a problem created by thinking about users as opposed to a mechanism on the way which is to think about value which actually ties together what you're doing along the way instead of just trying to solve the problem later so what am I talking about so first when you think about value value changes things it makes it pretty unique by company your value equation is very different than my value equation and when I said earlier I was talking about new metrics the push is really for you to think about what is value at our company do I know if my CEO brought me in a room and said what is the unit of value that is important to us that I am trying to work to increase would you have an answer and with your answer be the same as your team's wonder answered be the same as your CEO at least in my experience there's been many times where I didn't have that answer at scoop my team probably had a different answer than me and their team probably had a different answer than them and so understanding what value is for your company is incredibly important but at the at the business line it's not that different from a growth equation than what we looked at before the only thing to add is frequency right and why is frequency so important here although not groundbreaking acquisition activation our user metrics do we have people are they doing a thing and actually retention in frequency is really the equation of value are other people who are still doing a thing are doing it a lot or doing it as much as I would want to actually create value for me right otherwise you're just looking at the user so let's use a really basic example and if this seems ridiculously simple that's exactly what I'm going for let's say you have a store it could be digital you've been selling digital t-shirts if that's your game or you could be selling actual t-shirts and let's say you have 10 people who bought t-shirts two of them bought really nice $50 t-shirts and eight of them bought five-dollar t-shirts if you think about daily actives if you think about typical cohort charts that show retention of a user and your team or you win and said okay I'm gonna come up with a way to improve my retention I want six instead of five people to buy again in the next six months well if all six of them came on the right side of this slide you've done a really bad job investing in the place where you got to create the most value because actually the only thing that matters is making sure that the folks on the left maybe the only thing have a better have a good experience and come back and buy again right that they're gonna continue to invest at that value over time or that you understand that maybe all people who buy five-dollar t-shirts today will buy $100 t-shirts tomorrow but if you take it at face value unattended that you want to retain the users you forget what you're actually trying to contribute back to your company which is value and if I told you you can have a company where you sell a thousand things for 50 bucks or 10,000 things for five dollars you would pick the former if you were trying to maximize for value again I know that seems simple but I I have been and I'm an advisor to an e-commerce company I've thought about this it's cool but there are many times where we were like well how many people are still using and we forgot that the people who were still using we're mixtures that people who have tremendous value for us and not a lot of value for us and so my charger the thing I think we need to do differently in each company is think about what is really the unit of value at our company and perhaps more importantly how do we have the same simple charts metrics graphs that help us steer toward value how do you get a growth team in the room or a product team in a room and say we are trying to move this number we are trying to change this chart in this way not 50 charts not a crazy dashboard not a spreadsheet and really grok the same action that you grok when you look at a user retention cohort so I'll talk about how we've done that it's cube so this is this is a typical cohort chart if you if this looks familiar to you raise your hand all right and if you use this in your company raise your hand all right so let's say at least at least over 50% of you right are using it and what does this say this says I have some number of users in month one of a cohort and by month n I'm going to lose some percentage of that cohort and if you're lucky it flattens quickly I mean it drops off quickly it flattens fast and even bends back if you're super lucky and then in your next cohort you hope that it flattens a little bit earlier and a little bit higher than the one before and that more users from that cohort because they like your product or your marketing better or continuing to use and the problem is that this normalizes all users as if they're the exact same and so you could have said you could go in a room a great marketer can walk in a room with the CEO and say look at this chart we went from you know 60% 12-month retention to 65% 12-month retention and in that process you could have actually lost value to the company how the 5% you added were at the expense of perhaps more valuable users that you would have wanted to retain but you change the product in a way that helped the majority as opposed to helping the people who got the most value out of your products in the early days of scoop that's exactly what we were doing we just wanted to get more people to carpool and so we're more people when people will give money away we'll do promotions will be extremely focused on how do we get more people to carpool after a month and the next co earth and this one and what we found is we were doing a really good job spending our time retaining users who did not have long term value they didn't have long term value because they didn't use this frequently they didn't have long term value because when the when the you know when you when you had to come back to relieving some of those promotions and pricing changes or things like that they stopped using the product so you had sort of faked their retention because you had inflated what they were getting out of the product in the short term but we had made the charts look good so we thought we were doing well but then we said ok well why are we not actually seeing more volume why are the trip volumes not going up in the same way that the user volumes are and so what we found over time is actually this is what our dominant cohort chart looks like it's good it's if you take anyone on the entire company because we put this around the entire company and you said what is you know which of these two cohorts you look at or what does this mean it looks like this and if you've been in marketing for a while or growth or products this should look really weird to you and the reason I say that is it's unusual it's a strange way to portray a cohort so much though that I have spent considerable amounts of time fundraising cycle Astro fundraising cycle explaining to our investors why our cohorts look like this and not the one that they're used to but why does this matter what this in what's funny is this slide will tell you everything about why scoop is a meaningful business and one side on anything else about scoot for you to get why it can be a valuable business and that is then what this says is that if a cohort starts using scoot and they start using and taking trips in their first month as they scale in month two month three month four that cohort will take more trips in months two three four and five and so on then they do in their first month so the value which is for us a trip right when people take trips that's good for scoop we make money the value actually increases from a cohort over time despite the fact that during the same period of time we're losing you know decent chunks of our raw users so we might churn 20 30 40 percent I won't get too specific of users over some number of months but our value actually increases anyway and the reason why is actually really simple it's because the users who churn usually it's because they take a couple trips and they never come back I'm sure you guys have the same same Jamie gave me folks in the room probably know exactly I'm talking about you might even be in the gaming session right and that's because you pick it up you try it thirty Seconds you're out these are not valuable users but what we found is the people who keep using scoop after that initial wave retains their volume and repeat usage goes up and up and up and up they might use it eight times in the first month 15 times in the second month then they're going to 20 25 30 35 and now they're commuting to work with scoop every single day this has replaced their commute but instead of thinking about let's keep everybody one of the things I remember standing there having to tell my team is if the other person who's not gonna be able to use us every day and this isn't working for you and after one or two trips you stop using scoop great that's fine I'm gonna focus on the person who can use this 40 times a month that's where value gets created and it changes your strategy it leads you don't want to do long term valuable things and so I'll skip past this quickly when you think about retention mandate you need those cohorts you also need a metric most people when they talk about value you're gonna tell me oh but we have an LTV number right we have an AV number we know what our unit of value is that was probably determined by an analyst or somebody doing some sequel sequel query at some point in the last year and then it never really changed right it's like it's like that number that everyone knows is true but actually no one's checked up on it in here and you know oh if I inquire user I get X back that's my Northstar but you're never able to optimize it on a week to week or month to month basis which means when you're doing cohort analysis and I'll keep skipping ahead but you're doing cohort analysis for project-based work you don't know what you're trying to optimize so what do I mean by all this you ultimately have to have one metric I think that can capsule eights what value is to your company in a way that every single project or growth or growth initiative that you do can be evaluated against that metric if I move this metric I know I'm creating value for the company and it helps me understand all the way to how I financially model the business so for scoop that's trips for register user and what's funny about that is it's literally just this equation but like mapped out in math we register a user then they take a trip then they continue to take trips and they take more of them we calculate that as trips registered user but with every feature I launch every campaign every promotion every time we change price saying I can then say ok for this cohort what is their trending in terms of the trips they take on a per registered user basis so much so that our entire financial model for the last two fundraisers is all built on this assumption for for how the company is going to grow which means that I can sit in a room with a PM and a marketer and say the project that we're gonna work on is literally tied to how we think about the gross scale of the company as opposed to some metric that is tied to a metric as tied to a metric as tied to a metric where finances over here talking about actual revenue so a couple examples from Skoob I'll go quick one we built a feature called the shortlist and it's um what this means is typically for scoop you scheduled a trip before nine o'clock the night before and we match you automatically we do it all in one time at nine o'clock and then you go on your way and you carpool what this let people do is schedule past that carpool deadline what's interesting about this is we knew or he believed it wasn't actually going to affect Ryder user retention we had not found that that ability was going to keep people using scoop longer we knew that it was going to fill value demand or value gaps that we already had riders who wanted something and drivers who wanted something who were flying by each other because they didn't have this opportunity to match after our kind of construct it's not unlike empty seats on an airplane right if all I looked at is if passengers are coming back that's not that useful but if I look at how I make sure every plane is full now I'm actually creating value for me as and so what that led to and this is how we actually think about we literally evaluating project by projects we were able to look at how the next cohort had a higher peak to its value curve and it's because we actually found that if you calculated the trips to registered user frequency was going up retention was going up a little bit but between the two of those you were driving value because you had solved the gap where there was an intent on need but previously people weren't able to take advantage of it if we had been thinking strictly about what caused people to drop off we would never have thought about this feature because this feature built for people who were already generally pretty satisfied they were using it they were using it you know 10 20 30 times a month they were not going from 30 to 35 but they were going up to 30 you catch my drift versus say hey how do we get the people who take two trips not to not to drop off but what you found what we found out is that's incredibly hard it is really difficult to figure out why your product doesn't work for this group of people versus really thinking about why does it work for the people it works for and how do i 10x that final final example we're working on a feature right now which is simple it's basically removing a rider if I can match with two riders how do I take one rider out if that person is not convenient for me to pick up same exact thing goes and I think this is a nice example to end on if I'm a driver the incidence rate with which I will find myself in this situation oh we really happens when I use scoop often right if I only use cube a few times the chances that I'm in a carpool with two other people and I want to remove one it's pretty slim most people don't get here in their first five trips but over the course of time this starts to cause people to use scoop a little less to be a little bit less frustrated it can be a little bit more frustrated to cancel on an entire carpool instead of an individual person all of those are value detractors and so when we've worked on this feature we will look at how are we creating long-term value from these cohorts and even seeing existing cohorts inflects because of how this affects them but again we would have been distracted by looking at users and so 30 seconds I left oh and with my homework or my suggestion really because it's just me trying to share what's worked for us is figure out what your value cohort looks like it might look like ours it might look like a normal cohort but be a different axis get away from that single or my users continuing to use and actually infuse the idea of what value they give you and pair it with a single metric that one metric to rule them all so that every time you walk in a room with your teams you know what you're trying to effect whether that's finance sales marketing products or whomever else so hopefully that's helpful I know that when I've been in your shoes I'd like to at least have the one thing I get to write down that I can match to my team so hopefully you have that I'm all done thank you very much and appreciate listen to me [Applause] [Music] you