Jason Prowd ‘Unpacking product metrics’

Adrienne: Hello, and welcome to Prod Pod.  We’re a part of This is HCD.  We’re bringing all human-centred practitioners real and honest conversations about product management.  My name is Adrienne Tan and I’m the co-founder of Brainmates.  Today, we’re talking to Jason Prowd about establishing and measuring product successful in financial services.  Jason is a product leader at Morning Star.  Since 1984, Morning Star has been helping investors achieve their financial goals by developing powerful researcher data and investment management products.

Jason loves understanding and solving complex investor problems in simple, delightful ways.  He currently leads two businesses for Morning Star, direct to consumer in Australia, and the global advisor experience for Morning Star investment management.  Prior to his current role, he designed, built, and ran a direct customer wealth management product for Morning Star.  Before that, he worked at an investment research and funds management business, both on the product and the investment research side.  Hello, Jason.  

Jason: Hello, it’s great to be here.  

Adrienne: You’ve been very fortunate to have been tasked with creating a new product from scratch at Morning Star, called: Morning Star Next.  Tell us about that whole experience.  

Jason: Sure.  Inventing a new product is a great privilege, it’s a lot of fun but it’s a lot of work, too.  I joined Morning Star about three years ago and put together a team to tackle the opportunity.  Really, that was understanding what the business wanted to do, so the strategy, why it wanted to pursue a direct to consumer investment product.  Traditionally, we distributed investment products via financial advisors.  Then I had to work on taking a Power Point presentation essentially and turning that into a real product.  Really interesting work.  

Adrienne: Excellent.  Because you’re going direct to customer instead of direct to a business or investors, how is the business setup to allow you essentially to design and build a product?  

Jason: That’s a great question.  I was fortunate to have really good support from both my local boss, the MD of Australia at the time and my divisional manager.  Both of them were really keen on this idea, so they created the environment that allowed us to go and try new things, take a few risks.  We want to as a business be trying to be trying new things.  They really set that structure up for me and then let me get to work.

Adrienne: Excellent.  What were some of the more poignant changes that they had to make?  What was some of the key changes in the business to allow you to do that?  

Jason: Leaving me alone.  

Adrienne: Excellent.  We love that.  

Jason: A lot of it was trusting me enough to put together a strategy and a plan and then letting me get on with executing it.  I think that a lot of new businesses, particularly in existing businesses can wither on the vine if they’re not given space.  My direct manager and the local MD really let me run with the idea.  We had a direct team.  They all worked for me, were working principally just on that product, so we could go and do different things that would be a bit more difficult if you had to work within the constraints of an existing business.  

Adrienne: I love that.  I love that one of the biggest things that you can allow product people to do is really give them the space, the time to do it.  I think that’s really great for lots of larger organisations who typically want to control the process.  

Jason: I think it’s super hard, right?  The ideal way to run a business is, say, you’ve got a good idea, I trust you, go do it.  Now, that doesn’t actually work, as soon as you have a business with more than a couple of people, you need some processors around things.  We’re a listed business, so you need some structure, so you need to control the risk in certain ways.  In our business, we can try out a new business in Australia with ten people and that’s not a big risk.  We can control it by the size of the investment.  It makes sense that businesses want some structure before they let people go off and do things.  

Adrienne: Yes, that’s excellent.  There’s been a change.  Now, this is probably very exciting.  It’s not very often that we hear organisations make this kind of change, I don’t think, but you’ve put your product Morning Star Next on hold essentially.  Tell me about that, how did that come about?  How did you make that decision?

Jason: Sure, it wasn’t easy at all.  I think if you’ve poured a couple of years of your life into something, it’s sad if you have to change direction.  A good business has to sit there and weight up the opportunities in the market place.  Really, the opportunity cost of pursuing one opportunity over another.  When we’re looking out a few years, end of last year, mid last year and thought, “Okay, where can we have our teams focused?  

What are the biggest and brightest opportunities for our business?” We decided that direct-to-consumer on a wealth management product in Australia wasn’t it.  It made sense to pivot the team and have them focus on something else, which they’re now focused on the advisor experience for our global investment management business.  

Really tough.  If you’ve worked on a product and you make that kind of change, I’d be really disappointed if you weren’t sad.  When we sat down with the team and discussed it with them, people looked upset.  That makes sense.  I want people on my team who buy into the vision and mission of what we’re trying to do.  At the same time, I think letting things drag on isn’t a great strategic decision either.  It was good that we were able to make a swift decision, pivot, and pursue a new opportunity.  

Adrienne: I guess that goes around the whole concept about establishing some metrics before you build and then measuring them throughout the time the product is in market.  Did you do that before you start?  Did you hold yourself accountable to those numbers?  I know numbers are really emotionally, you look at them, you go, hey, I meant to hit this, but I haven’t and here are the reasons, I’ll keep going.  Talk to me about how you did that, before you made the product and how you landed where you are now with the product.  

Jason: Metrics are super important I think for any business to build trust; you have to be really transparent about what you’re trying to achieve and what’s realistic.  I think I’ll share a story from my time as being a research analyst.  When we covered SX companies, we were making forecasts about what we thought a company was going to do in the future.  One thing I really like doing was what was called a premortem, so you’d write before the fact what you thought was going to happen.  

You couldn’t see a company’s results and say, “Hey, I always thought that was going to happen.” You had to put pen to paper and say, “Here’s what I think.  The balance and probabilities is the likely outcome for X, Y, Z company.” I applied that same logic for thinking about our new product at Morning Star.  We’re an investment business, so there are lots of analytical people in the business.  I knew it was going to be really important early on to establish credibility in the organisation, showing that we were going to be really transparent in how we measured things, and setting some ambitious, yet, realistic ways of measuring product success.  

For an investment management product, that’s not actually that complicated, it’s really how many clients are you acquiring?  How much are they choosing to invest?  How much is it costing you to acquire those clients?  Then once you’ve acquired them, how happy are they?  We had basically four different metrics and we set out a range of outcomes when we set up the product.  We tracked them the whole way through.  

Adrienne: Right.  At what point do you decide that enough is enough?  It’s really a very difficult question to answer, I guess, but…

Jason: Yes, I think it’s very difficult.  I think the difficult with metrics is, it’s very easy to gravitate to one number.  I think you need to think about measuring things quantitatively and qualitatively.  Quant is easy in lots of ways, capturing data now is super easy.  Strategy, though, is about blending that quantitative data and the qualitative insight and making a trade-off, right?  If you think about it, strategy is simply just, what are we doing and why?  I think for us with Next, the decision we made was, we looked at the potential upside if we kept pursuing it in its current form over the next three to five years.  

We looked at the potential upside of pivoting and doing something else.  It was less about Morning Star Next not working out exactly as we’d hoped.  It was more about comparing it to another opportunity and seeing that that was orders of magnitude bigger.  For us, a very successful outcome for Morning Star Next over a five to ten-year period was tens of millions of dollars of revenue.  That would be a fantastic outcome from zero.  

Our investment management business always generates over 100 million dollars of revenue U.S.  a year.  Making small changes there can have a much, much bigger impact on our overall business.  For us, it was simply weight up the opportunity cost, I guess.  

Adrienne: Got it.  That leads to another point, though, it’s when you’ve got these new product ideas, especially in really large organisations, they’re never going to compare, it’s always going to take longer to be able to catch up to your existing product.  How then do you justify making these innovative products?  How do you do that?  

Jason: I think that’s the tough work of strategy and one – it’s a good question for any CEO of any business, the way we’ve decided to think about things is, we really classify our businesses in different categories, this allows us to make different types of investment decisions.  Businesses where we’re seeding them, so it might be something new that may or may not work out, and the time frame is going to be long.  It’s going to be super uncertain.  Then we’re going to have businesses that we say are in growth mode.  Things that we know the customer proposition is working, it’s all starting to come together and really what you need to do is just let them run and give them some more capital to let the business grow.  

Then you have businesses that are more in harvest mode, so businesses that might have been really successful or products that have been really successful in the past, but you can see, over the next five/ten years, they’re not going to be as successful in the future, maybe they’re not where the market needs your company to be.  

Those businesses can really help fund those two other categories.  I think you need to get that right at the board/CEO level.  Otherwise, no one has any incentive to try out new things that may or may not work.  Most professional managers will pick projects that have very certain outcomes and don’t take on a lot of career risk.  

Adrienne: Yes, that’s a really good point.  It’s the balancing of your portfolio really, of your investments across different kinds of products at different stages in their lifecycle.  This leads me to discussing your current products, which I suspect is more established.  Firstly, how do you go from working in something brand-new, to something more established?  What are the kind of changes in your day-to-day work life as a product manager, going from fairly brand-new to established product?  

Jason: Sure, I’ll expand on the opportunities a bit.  In our direct to consumer space, so how we help individual investors in Australia at the moment, we have an existing research data product, where you can go and get some free research, and then you can sign up for a subscription.  That’s being rebuilt at the moment.  In a sense, the proposition is pretty clear, we know there’s a market there for that type of product.  What work we’re focused on at the moment is really re-platforming what we have.  

That is different but I think you can apply some of the same types of thinking you do to a new product.  You want to be interactive you want it to be based on data, you want to listen to your customers, all of that good stuff, I think you can apply to an existing product.  In the advisor space, while the business is established and over the last five/ten years, we’ve done a nice job building an investment management business that helps financial advisors.  

The way we serve those advisors beyond given them great investment products is quite nascent at the moment.  In a sense, I am working on something entirely new again, because that advisor experience for our core market, so that’s the UK, the U.S., here and South Africa is only just getting started.  The team and I are going to be using very, very similar ideas and even insights we got from working directly with consumers and applying it to how we build out that service experience for our advisor clients.  

Adrienne: In terms of measurement and metrics, do you think it’s any different to your initial foray into the B2C space?  Or are they similar?  

Jason: Yes, I think the things you want to measure are a little bit different.  The first thing I’d say about metrics is, I like dividing goals and KPIs.  Goals, as I see it, are what you want to get done.  KPIs are things you measure along the way to see if you’re tracking towards that goal.  Within that context, you need both longer-term ambitions, things that will show you a product or business has been successful over time, but then you’re going to need indicators along the way.  Ideally, what you need are leading indicators.  

What I see a lot of in business is lagging indicators.  A good example of that, is tracking something like revenue.  Revenue largely tells you about all growth in revenue sale fall.  It tells you largely about what’s happened in the past, right?  It’s likely driven by products that have been built in the past.  It’s been driven by customer relationships in the past.  A business can have great revenue growth, but that might not tell you in any one year what’s going to happen over five years.  You need earlier indicators, you need to make sure you measure them, as well.  Now, of course, you can’t just measure qualitative things and say that leads to product success.  

Ultimately, a business is a balance between commercial success and customer success.  If the business wins too much, the customers get really angry and leave.  If the customers win too much, the business goes under.  I think you need to balance having metrics that measure both of those factors.  We’ve spent a lot of time kicking around ideas for advisor experience, and that’s sort of TBC exactly what we’ll measure, but we’ve got ideas around what makes sense for investment management business to measure long-term.  Then, also, some ideas around what are good leading indicators that are worth measuring.  

I think the other thing to remember about metrics is to not be too obsessive about them.  I think I see a lot of data where what people are forgetting is the analysis.  I can pull up some spreadsheet and show you some numbers, what’s important is what’s driving those numbers.  My take is, most businesses over metric themselves.  They’ll have 15/20/30 different things that they claim tell you how a business is working.  Often, that’s driven by managers wanting to have some kind of number that’s got a green tick against it.  The most complex businesses in the world only have a few metrics that indicate whether they’re going well or not.  

Financial services, people would hold up a bank and say, “Come off it, bank, it’s super complicated.” They do all of these different things, they’ve got tens of thousands of staff, how can you measure that business with a few simple metrics?  It will be like, well, okay, what does come off bank actually do?  It sells primarily two products, home loans and credit cards.  How do they make money?  

They make money by the spread between what they borrow money at and what they lend it to you at.  Then the other important thing for a bank is, how likely is it that the people that have borrowed this money are going to pay it back?  Even something as complex as a bank, like, if you don’t get the analysis of those three metrics, it doesn’t matter what the other fifty are.  I think it’s important to really focus in on metrics that tell you something useful and hopefully, tell you something leading, not lagging.  

Adrienne: What might be – I mean, I love the whole leading/lagging indicators.  I think that we don’t actually pay enough attention to any of those indicators.  What might be some leading indicators for financial services?  Not necessarily Morning Star, but just financial services in general.  

Jason: Yes, lots of normal ones that apply across product categories.  Customer adoption, customer satisfaction, things that might not translate into lots of customers today, but the further you go towards how new people are using your product.  The big issue any existing product has, right, is that they may look like they’re very profitable and very successful, but actually are doing a really bad job at winning over new customers.  What’s important is, as a business grows, the marginal customer they acquire is les excited about what you do.  

When you have a startup and it’s the first 100, whatever, customers, they’re the people who really love what you’re doing.  As you get bigger and bigger and bigger, each marginal customer is less likely to like what you’re doing because you’ve done a really good job at winning over all the ones that love what you’re doing.  It’s really important to understand, if the product is actually satisfying those people, and I think that can tell you a lot.  I like to talk about thinking about your pre-customers, or those who are not yet your client.  

Existing businesses spend a lot of time, and we’re on a podcast, I’m going to pretend that I’m looking behind me, I think they spend a lot of time looking at their, internally, looking at their existing customer and saying, we have to keep them very delighted.  I think that’s true if you have a dominant market position and have most of the customers already signed up.  That’s not true of most businesses.  Most businesses aren’t Telstra.  If you’re Telstra, what you need to worry about is keeping all of your mobile phones.  

Adrienne: Retention.  

Jason: Retention.  That matters.  If you’re at Maysim, you need to worry about signing up new people.   It doesn’t matter, it matters how happy your existing clients are, but you want to worry about those marginal ones.  That would be one I’d focus on for sure.  

Adrienne: Excellent.  Thank you.  You’re a product leader, do your product managers pay attention to the numbers, as well, or is it more of a senior role to pay attention to numbers?  

Jason: I’m not sure how any product person of any kind can do their job without having a sense of how they’re going.  That’s really what metrics tell you, right?  They tell you how I’m going?  They’re a way of quantifying something about what you’re doing, so I think that’s not about seniority.  That’s about the role itself.  I do think it’s incumbent on the more senior folk in the organisation to set clear goals and how to measure them for people.

I think it’s very hard to expect someone super junior in an organisation who’s new in an organisation to come up with every way to measure their product.  That’s absurd.  The role of leadership is to connect up the mission and goals of the company with someone’s specific role, but it’s equally on that individual contributor to understand how they’re going.  That goes for beyond product.  

Adrienne: Yes, absolutely.  I completely agree.  Part of the product manager role is to understand and be excited about how your product is performing.  Once you do this reporting, from your point of view, who do you report these metrics, these numbers to, what do you do about them?  

Jason: I’m a big believer in product transparency to the extent that that’s possible within your organisation.  My general view would be, be more open than closed.  You need to report these metrics to lots of people.  I think it is absolutely critical that the team themselves understand why metrics are being measured and what they are and how they’re going.  That’s amongst the team.  Then, often, really what metrics are about is reporting sideways and up.  They’re about keeping your stakeholders across what you’re doing.  I think open transparent and sensible metrics help drive trust.  Trust lets you get your job done.  We usually develop some kind of dashboard for any product that we have.  Then we use that same dashboard for both reporting to stakeholders and the team, so it’s all consistent.  I think that’s really helpful.  

Adrienne: Excellent.  A lot of our product managers that I come in contact with tell me that it’s very hard to get the data to produce a dashboard, to produce a report.  How do you go about it?  Especially for new products?  That’s why people say they can’t measure how their product is performing because they can’t get access to the data, what are your opinions?  

Jason: I think that’s probably true.  I think it’s much easier to get the right kind of data if you think about it up front.  I consider it a table-stakes part of building a product.  If you think about a digital product, right, it’s table stakes that a website loads in half a second or whatever it is.  It’s table stakes to me that a contemporary product has the ability to measure itself, built in from the start.  I think the issue often occurs that people build things and then after the fact they’re like, “It would be great if I could measure X.” The problem is, often you need to have thought of that up front.  I’d say it’s in the inception stage.  

I also just think that for any product out there, back to my earlier point about people trying to measure too many different things, you really need to distil down what are those couple of things that might swing the outcome of your product and go about measuring them.  Often, it’s a question of resources, it’s a question of how do I go about measuring this?  Start with the big metrics that matter.  If I’ve looked at hundreds of financial models, typically, it’s only a couple of things that swing the outcome.  That’s true of products.  Narrow down the focus and then get it implemented.  I sympathise.  

Adrienne: Excellent.  Well, thank you very much for your time.  

Jason: Of course, it’s been great.  

Adrienne: I always enjoy talking to you.  

Jason: Indeed.  

Adrienne: Thank you for listening to Prod Pod.  If you want to learn more about the wonderful shows on This is HCD, we’ve got others, such as the Power of Ten with Andy Polaine, Bringing Design Closer with Gerry Scullion, and Ethno Pod with Dr.  John Curran.  Please visit: thisishcd.com, where you can sign up to our newsletter, join our Slack channel, connect with other human-centred practitioners around the world.  Thank you for listening and speak to you next time.  

End of Audio